-- Logs begin at Fri 2022-09-23 13:06:21 UTC, end at Fri 2022-09-23 14:26:54 UTC. -- Sep 23 13:41:51.102929 dv-cinder-857434-4-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 23 13:41:51.110365 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 23 13:41:51.114747 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: *** Starting uWSGI 2.0.18-debian (64bit) on [Fri Sep 23 13:41:51 2022] *** Sep 23 13:41:51.114747 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 23 13:41:51.114747 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 23 13:41:51.114747 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: nodename: dv-cinder-857434-4-iscsi Sep 23 13:41:51.114747 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: machine: x86_64 Sep 23 13:41:51.114747 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: clock source: unix Sep 23 13:41:51.114747 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: pcre jit disabled Sep 23 13:41:51.114747 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: detected number of CPU cores: 4 Sep 23 13:41:51.114747 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: current working directory: / Sep 23 13:41:51.114747 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: detected binary path: /usr/bin/uwsgi-core Sep 23 13:41:51.114747 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: your processes number limit is 47680 Sep 23 13:41:51.114747 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: your memory page size is 4096 bytes Sep 23 13:41:51.114747 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: detected max file descriptor number: 2048 Sep 23 13:41:51.114747 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: lock engine: pthread robust mutexes Sep 23 13:41:51.114747 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: thunder lock: enabled Sep 23 13:41:51.114747 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 23 13:41:51.114747 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 23 13:41:51.153020 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: Python main interpreter initialized at 0x55dbe90cb9c0 Sep 23 13:41:51.153020 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: python threads support enabled Sep 23 13:41:51.153020 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: your server socket listen backlog is limited to 100 connections Sep 23 13:41:51.153020 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: your mercy for graceful operations on workers is 90 seconds Sep 23 13:41:51.153388 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: mapped 403077 bytes (393 KB) for 2 cores Sep 23 13:41:51.153464 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: *** Operational MODE: preforking *** Sep 23 13:41:51.153708 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: *** uWSGI is running in multiple interpreter mode *** Sep 23 13:41:51.153708 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: spawned uWSGI master process (pid: 81553) Sep 23 13:41:51.153914 dv-cinder-857434-4-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 23 13:41:51.154503 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: spawned uWSGI worker 1 (pid: 81568, cores: 1) Sep 23 13:41:51.154837 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: spawned uWSGI worker 2 (pid: 81569, cores: 1) Sep 23 13:41:51.155851 dv-cinder-857434-4-iscsi devstack@keystone.service[81553]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 23 13:41:55.519938 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 23 13:41:55.519938 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: from cryptography.utils import int_from_bytes Sep 23 13:41:55.520687 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 23 13:41:55.520687 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: from cryptography.utils import int_from_bytes Sep 23 13:41:55.520819 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 23 13:41:55.520819 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: from cryptography.utils import int_from_bytes Sep 23 13:41:55.520819 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 23 13:41:55.520819 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: from cryptography.utils import int_from_bytes Sep 23 13:41:55.571336 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 23 13:41:55.571521 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 23 13:41:55.571603 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 23 13:41:55.571677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 23 13:41:55.571744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] command line args: [] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 23 13:41:55.571806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] command line args: [] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 23 13:41:55.571983 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 23 13:41:55.572076 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 23 13:41:55.572152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ================================================================================ {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 23 13:41:55.572227 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ================================================================================ {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 23 13:41:55.572378 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] admin_token = **** {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.572460 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] admin_token = **** {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.572626 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] config_dir = [] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.572708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] config_dir = [] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.572812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.572893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.572973 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] config_source = [] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.573040 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] config_source = [] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.573155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] debug = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.573238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] debug = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.573355 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.573611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.573747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.573863 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.573968 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.574070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.574168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.574259 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.574391 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.574497 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.574598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] list_limit = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.574696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] list_limit = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.574798 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] log_config_append = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.574930 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] log_config_append = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.575071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.575171 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.575263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] log_dir = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.575360 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] log_dir = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.575459 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] log_file = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.575557 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] log_file = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.575557 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.575737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.575801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.575916 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.576072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.576163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.576285 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.576360 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.576481 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.576565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.576684 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.576757 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.576875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.576996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.577120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.577249 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.577367 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.577441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.577634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.577713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.577875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.577998 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.578110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.578201 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.578384 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.578505 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.578632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.578768 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.578906 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.578998 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.579403 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.62/identity {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.579503 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.62/identity {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.579576 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] publish_errors = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.579691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] publish_errors = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.579804 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.579881 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.580035 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.580109 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.580215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.580338 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.580459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.580547 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.580684 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.580765 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.580918 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] standard_threads = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.581005 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] standard_threads = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.581147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.581221 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.581390 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.581463 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.581563 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.581648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.581724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] use_journal = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.581797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] use_journal = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.581899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] use_json = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.581980 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] use_json = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.582057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] use_stderr = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.582132 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] use_stderr = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.582230 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] use_syslog = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.582380 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] use_syslog = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.582484 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.582565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 23 13:41:55.582737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.582818 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.582977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.583050 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.583232 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.583312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.583458 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.583530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.583704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.583829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.583941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.584024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.584187 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.584262 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.584418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.584493 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.584663 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.584745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.584907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.585030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.585155 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.585230 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.585391 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.585528 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.585640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.585780 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.585878 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.586016 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.586129 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.586268 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.586445 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.586542 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.586618 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.586778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.586877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.587032 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.587137 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.587285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.587387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.587529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.587626 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.587764 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.587883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.588015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.588146 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.588271 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.588406 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.588521 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.588658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.588784 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.588893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.589013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.589128 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.589241 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.589391 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.589463 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.589588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.589690 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.589838 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.589921 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.590088 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.590164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.590320 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.590472 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.590581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.590669 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.591092 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.591183 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.591304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.591376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.591443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.591519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.591594 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.591662 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.591809 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.591885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.592044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.592125 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.592277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.592358 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.592505 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.592616 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.592869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.592978 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.593108 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.593210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.593345 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.593446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.593580 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.593685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.593821 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.593904 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.594051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.594132 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.594287 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.594425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.594543 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.594614 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.594766 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.594851 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.595000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.595090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.595247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.595324 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.595476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.595555 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.595711 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.595826 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.595946 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.596029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.596185 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.596259 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.596418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.596492 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.596652 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.596724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.596879 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.596955 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.597124 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.597199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.597369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.597465 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.597602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.597690 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.597837 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.597917 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.598098 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.598181 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.598368 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.598451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.598581 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.598656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.598818 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.602291 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.602291 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.602439 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.602439 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.602573 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.602643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.602721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.602790 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.602858 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.602930 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.602998 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.603076 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.603151 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.603219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.603285 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.603351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.603418 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.603418 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.603624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.603693 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.603760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.603826 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.603892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.603967 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.604036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.604102 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.604169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.604235 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.604301 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.604381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.604457 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.604534 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.604609 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.604678 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.604744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.604811 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.604877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.604951 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.605025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.605094 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.605161 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.605161 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.605284 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.605359 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.605429 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.605495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.605562 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.605628 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.605694 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.605768 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.605838 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.605905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.605983 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.606052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.606118 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.606192 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.606262 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.606382 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.606455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.606523 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.606589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.606668 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.606744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.606820 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.606889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.606961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.607030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.607104 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.607174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.607240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.607307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.607373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.607450 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.607527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.607597 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.607660 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.607726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.607868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.607945 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.608122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.608884 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] auth.external = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.608968 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.609120 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.609198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] auth.external = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.609404 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.609507 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.609698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.609784 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.609931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.610007 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] auth.password = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.610162 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] auth.password = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.610239 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] auth.token = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.610415 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] auth.token = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.610496 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.610657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.610734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.610894 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.610971 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.611127 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.611204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.611361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.611436 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.611608 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.611685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.611841 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.611917 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.612072 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.612148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] credential.caching = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.612307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] credential.caching = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.612384 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.612553 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.612631 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.612800 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.612876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.613045 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.613120 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.613285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.613364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.613520 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.613598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.613768 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.613845 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.614030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.614117 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.614268 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.614369 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.614527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.614603 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.614782 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.614861 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.615058 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.615134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.615301 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.615377 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.615555 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.615633 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.615794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.615869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.616040 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] federation.caching = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.616124 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.616277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] federation.caching = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.616360 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.616515 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.616593 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.616759 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.616835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.616989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.617067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.617220 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.617302 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.617462 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.617538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.617696 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.617773 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.617944 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.618022 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.618178 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.618254 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.618450 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.618529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.618705 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.618789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.618951 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] identity.caching = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.619035 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.619203 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.619298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] identity.caching = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.619454 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.619523 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.619698 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.619789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.619933 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.620013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.620167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.620242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.620404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.620487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.620632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.620708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.620869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.620946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.621115 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.621200 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.621349 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.621424 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.621583 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.621659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.621812 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.621896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.622042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.622118 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.622283 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.622415 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.622553 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.622643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.622798 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.622876 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.623046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.623122 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.623288 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.623372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.623547 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.623629 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.623808 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.624514 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.624582 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.624690 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.624833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.624942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.625076 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.625181 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.625306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.625422 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.625541 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.625668 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.625780 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.625911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.626017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.626163 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.626268 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.626421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.626524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.626673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.626770 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.626923 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.627002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.627425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.627510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.627673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.627752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.627922 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.628007 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.628158 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.628235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.628398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.628476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.628625 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.628702 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.628861 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.628938 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.629096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.629173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.629335 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.629412 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.629567 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.629642 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.629801 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.629877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.630048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.630126 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.630426 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.630509 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.630661 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.630748 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.630893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.630972 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.631140 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.631218 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.631388 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.631466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.631624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.631702 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.631857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.631934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.632093 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.632169 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.632325 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.user = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.632403 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.user = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.632560 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.632636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.632816 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.632893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.633048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.633125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.633292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.633368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.633538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.633623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.633785 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.633861 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.634016 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.634091 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.634245 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.634343 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.634503 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.634581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.634738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.634815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.634974 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.635062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.635202 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.635280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.635446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.635522 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.635688 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.635773 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.635935 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.636026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.636176 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.636252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.636419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.636504 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.636649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.636726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.636894 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.636972 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.637129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.637214 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.637363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.637440 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.637600 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.637676 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.637833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.637916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.638077 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.638154 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.638321 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.638423 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.638578 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.638667 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.638830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.638908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.639061 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.639136 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.639300 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.639449 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.639558 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.639636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.639793 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.639910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.640056 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.640229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.640310 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.640477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.640555 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.640712 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.640788 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.640941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.641024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.641176 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.641253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] resource.caching = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.641416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] resource.caching = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.641492 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.641661 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.641738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.641898 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.641975 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.642130 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.642206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.642398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.642480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.642636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.642712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.642870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.648137 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.648137 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.648325 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.648385 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.648442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.648498 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] role.caching = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.648566 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.648566 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] role.caching = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.648701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] role.driver = sql {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.648701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.648825 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] role.driver = sql {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.648825 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.648947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.649023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.649093 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.649151 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.649214 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.649283 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.649351 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.649427 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.649498 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.649567 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.649635 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.649703 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.649778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.649849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.649918 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.649918 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.650040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.650040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.650172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.650243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.650310 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.650448 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.650523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.650592 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.650673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.650745 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.650814 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.650882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.650950 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.651018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.651089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.651089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.651205 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.651273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.651341 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.651408 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.651484 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.651556 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.651624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.651624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.651736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.651804 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.651880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.651880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.651996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.652064 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.652132 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.652200 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.652276 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.652346 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.652414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.652414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.652526 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.652601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.652672 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.652740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.652808 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.652871 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.652938 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.653013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.653083 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.653151 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.653277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.653398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.653473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] token.caching = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.653545 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] token.caching = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.653616 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.653684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.653752 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.653820 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.653884 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.653959 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.654029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.654102 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.654183 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.654242 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.654298 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.654420 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.654495 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.654564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.654633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.654702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.654771 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.654847 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.654918 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.654987 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.655543 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.655614 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.655683 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.655760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.655830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.655899 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.655966 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.655966 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.656098 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.656165 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.656374 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.656461 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.656635 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] database.connection = **** {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.656722 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.656883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.656961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.657132 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.657210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] database.connection = **** {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.657376 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.657454 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.657617 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.657700 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.657854 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.657931 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.658093 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.658171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.658339 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.658418 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.658585 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.658661 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.658805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.658878 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.659058 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.659134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.659306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.659377 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.659541 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.659618 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.659780 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.659855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.660012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.660089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.660255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.660332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.660486 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.660562 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.660724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.660799 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.660953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.661030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.661122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81569) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 23 13:41:55.661197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.661411 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 23 13:41:55.661578 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81568) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 23 13:41:55.690843 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:55.691071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:55.691299 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:55.691381 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:55.691536 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:55.691756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:55.699724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:55.700161 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:55.729208 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:55.729410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:55.729516 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:55.729637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:55.791387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:55.791544 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:55.839955 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:55.840188 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:55.840797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:55.841031 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.000350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.000577 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.000733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.000847 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.000970 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.001054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.001195 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.001318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.001407 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.001481 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.001620 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.001835 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.025405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.025641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.025734 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.025868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.025950 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.026096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.026175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.026316 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.026458 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.026538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.026599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.026759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.034460 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.034460 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.045298 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.045397 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.058063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.058292 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.058408 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.058543 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.058624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.058772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.058863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.059047 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.059139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.059212 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.059282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.059427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81569) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.059519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.059643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81568) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 23 13:41:56.081245 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.081455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 23 13:41:56.081538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.081695 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 23 13:41:56.081805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.082013 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 23 13:41:56.082129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.082293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 23 13:41:56.082400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.082568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.082666 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.082795 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.082874 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.083071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.083166 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.083313 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.092800 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.092970 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.093075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.093250 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.093356 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.093517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.093600 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.093798 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.100589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81569) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.100743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81568) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.103518 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81569) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.103639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81568) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.103806 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.103930 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.104089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.104216 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.104373 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.104502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.104653 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.104787 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.104956 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.105038 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.105232 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.105310 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.105509 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.105595 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.105782 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.105860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.106058 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.106136 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.106347 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.106429 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.130379 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.130482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.131836 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81568) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.131926 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81569) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.132119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.132207 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.136771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81568) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.136870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81569) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.137053 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.138171 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.138256 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.138324 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.143644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81568) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.143814 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81569) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.143925 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.144101 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.149574 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81568) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.149849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.149941 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81569) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.150132 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.150233 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.150444 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.150529 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.150726 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.150809 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.150998 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.151081 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.151268 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.151348 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.151534 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.151628 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.151804 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.167028 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.167290 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.167373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.167578 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.171845 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81568) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.172206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81569) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.175148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.175446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.175527 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.175804 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.179654 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81568) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.180005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81569) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.182777 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81568) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.183162 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81569) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.185987 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.186921 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.187131 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.187207 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.187429 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.187524 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.187703 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.187785 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.187978 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.188058 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.188330 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.188621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.203711 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81568) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.204021 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.204271 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.204536 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.204667 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81569) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.204958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.205224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.205486 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.211385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.212377 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.212925 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.213980 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.214474 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.214750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.215531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.215807 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.217438 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81568) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.217844 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.218129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.218433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.218791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81569) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.218871 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.219080 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.219370 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.219649 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.220046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.229885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81568) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.230168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.230467 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.230748 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.231124 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.231203 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81569) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.231413 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.231506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.231699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.231780 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.232016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.232399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.232701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.232979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.247557 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81568) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.249676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81569) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.251144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81568) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.252879 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81569) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.260052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.260052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.260190 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.260261 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81568) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.260331 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.260414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.260414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.260549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81569) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.260549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.260549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.264259 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.266229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81568) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.266345 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.267794 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81569) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.269107 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.269502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.269786 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.270063 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.270723 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.271122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.271407 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.271682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.277846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81568) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.278138 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.278451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.279458 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81569) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.279750 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.280624 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.285913 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81568) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.286188 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.286495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.291133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.291133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.291133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.291133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.291423 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81569) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 23 13:41:56.291484 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.291568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.291642 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.291713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.291784 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.291854 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.291933 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.292005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.292075 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.292140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.292209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.292288 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.292360 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81568) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.292431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.292431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.292431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.292431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.292431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.292431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.292773 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81569) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 23 13:41:56.325553 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55dbe90cb9c0 pid: 81568 (default app) Sep 23 13:41:56.325669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55dbe90cb9c0 pid: 81569 (default app) Sep 23 13:41:56.326484 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816408b4-9b46-4029-a8c9-aa34d6187a56 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:41:56.326771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816408b4-9b46-4029-a8c9-aa34d6187a56 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:41:56.327583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816408b4-9b46-4029-a8c9-aa34d6187a56 None None] PATH_INFO: `/v3` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:41:56.331714 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1/1] 10.222.0.62 () {50 vars in 791 bytes} [Fri Sep 23 13:41:56 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:03.822919 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601f3e2e-21ee-45b1-8d50-6977aada3e61 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:03.823452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601f3e2e-21ee-45b1-8d50-6977aada3e61 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:03.823452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601f3e2e-21ee-45b1-8d50-6977aada3e61 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:03.828474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1/2] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:03 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:03.834283 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd1d2b7-f403-424d-afc0-b5b1d4b95bbc None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:03.834757 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd1d2b7-f403-424d-afc0-b5b1d4b95bbc None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:03.834849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd1d2b7-f403-424d-afc0-b5b1d4b95bbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:03.905968 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: INFO dbcounter [None req-ccd1d2b7-f403-424d-afc0-b5b1d4b95bbc None None] Registered counter for database keystone Sep 23 13:42:03.925101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG oslo_db.sqlalchemy.engines [None req-ccd1d2b7-f403-424d-afc0-b5b1d4b95bbc None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81568) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 23 13:42:03.931390 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writer thread running {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 23 13:42:04.116781 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG passlib.handlers.bcrypt [None req-ccd1d2b7-f403-424d-afc0-b5b1d4b95bbc None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81568) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 23 13:42:04.116781 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG passlib.handlers.bcrypt [None req-ccd1d2b7-f403-424d-afc0-b5b1d4b95bbc None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81568) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 23 13:42:04.169703 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-ccd1d2b7-f403-424d-afc0-b5b1d4b95bbc None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:04.171605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-ccd1d2b7-f403-424d-afc0-b5b1d4b95bbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:04.300697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 2/3] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:03 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 467 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Sep 23 13:42:04.782912 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601f3e2e-21ee-45b1-8d50-6977aada3e61 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:04.854009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: INFO dbcounter [None req-601f3e2e-21ee-45b1-8d50-6977aada3e61 None None] Registered counter for database keystone Sep 23 13:42:04.875972 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG oslo_db.sqlalchemy.engines [None req-601f3e2e-21ee-45b1-8d50-6977aada3e61 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81569) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 23 13:42:04.882245 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writer thread running {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 23 13:42:04.999931 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601f3e2e-21ee-45b1-8d50-6977aada3e61 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:05.004420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f264036-f2ce-4fe9-96b0-efdea14e8a76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:05.005032 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f264036-f2ce-4fe9-96b0-efdea14e8a76 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:05.005310 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f264036-f2ce-4fe9-96b0-efdea14e8a76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:05.005575 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f264036-f2ce-4fe9-96b0-efdea14e8a76 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:05.013595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f264036-f2ce-4fe9-96b0-efdea14e8a76 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:05.065014 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: WARNING keystone.common.rbac_enforcer.enforcer [None req-2f264036-f2ce-4fe9-96b0-efdea14e8a76 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 23 13:42:05.065307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f264036-f2ce-4fe9-96b0-efdea14e8a76 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:05.074436 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 2/4] 10.222.0.62 () {58 vars in 1324 bytes} [Fri Sep 23 13:42:04 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 23 13:42:05.081026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd1d2b7-f403-424d-afc0-b5b1d4b95bbc None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:05.098877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd1d2b7-f403-424d-afc0-b5b1d4b95bbc None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:05.103325 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b6baf7-f3a3-4cc0-96e6-a9f805273dfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:05.104407 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b6baf7-f3a3-4cc0-96e6-a9f805273dfa admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:05.104683 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b6baf7-f3a3-4cc0-96e6-a9f805273dfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:05.104963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b6baf7-f3a3-4cc0-96e6-a9f805273dfa admin admin] PATH_INFO: `/v3/projects/e15d4d7f53c543e88cdf9075fbadb3c8` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:05.106569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b6baf7-f3a3-4cc0-96e6-a9f805273dfa admin admin] RBAC: Authorizing `identity:get_project(project_id=e15d4d7f53c543e88cdf9075fbadb3c8)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:05.159007 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: WARNING keystone.common.rbac_enforcer.enforcer [None req-f6b6baf7-f3a3-4cc0-96e6-a9f805273dfa admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 23 13:42:05.159347 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b6baf7-f3a3-4cc0-96e6-a9f805273dfa admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:05.163880 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 3/5] 10.222.0.62 () {56 vars in 1207 bytes} [Fri Sep 23 13:42:05 2022] GET /identity/v3/projects/e15d4d7f53c543e88cdf9075fbadb3c8 => generated 344 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:07.302581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe690acc-d719-40f6-9092-367272052fd6 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:07.303093 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe690acc-d719-40f6-9092-367272052fd6 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:07.303179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe690acc-d719-40f6-9092-367272052fd6 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:07.307232 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 3/6] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:07 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:42:07.313407 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c431732d-7d27-4613-8e47-16203a57ea6a None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:07.313819 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c431732d-7d27-4613-8e47-16203a57ea6a None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:07.313957 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c431732d-7d27-4613-8e47-16203a57ea6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:07.333611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c431732d-7d27-4613-8e47-16203a57ea6a None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:07.335169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c431732d-7d27-4613-8e47-16203a57ea6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:07.342932 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 4/7] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:07 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Sep 23 13:42:07.605177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe690acc-d719-40f6-9092-367272052fd6 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:07.617680 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe690acc-d719-40f6-9092-367272052fd6 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:07.622146 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa495a88-e1c0-4093-9851-5d104979bac7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:07.622787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa495a88-e1c0-4093-9851-5d104979bac7 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:07.623125 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa495a88-e1c0-4093-9851-5d104979bac7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:07.623412 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa495a88-e1c0-4093-9851-5d104979bac7 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:07.631954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa495a88-e1c0-4093-9851-5d104979bac7 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:07.633711 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa495a88-e1c0-4093-9851-5d104979bac7 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:07.642410 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 4/8] 10.222.0.62 () {58 vars in 1324 bytes} [Fri Sep 23 13:42:07 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 23 13:42:07.647626 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c431732d-7d27-4613-8e47-16203a57ea6a None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:07.657419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c431732d-7d27-4613-8e47-16203a57ea6a None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:07.661721 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65759056-d4c2-4cce-ac29-0f2d85f28a93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:07.662359 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65759056-d4c2-4cce-ac29-0f2d85f28a93 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:07.662649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65759056-d4c2-4cce-ac29-0f2d85f28a93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:07.662921 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65759056-d4c2-4cce-ac29-0f2d85f28a93 admin admin] PATH_INFO: `/v3/users/2ea3af238714467ab48e451a5210c19f` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:07.664221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65759056-d4c2-4cce-ac29-0f2d85f28a93 admin admin] RBAC: Authorizing `identity:get_user(user_id=2ea3af238714467ab48e451a5210c19f)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:07.666037 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65759056-d4c2-4cce-ac29-0f2d85f28a93 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:07.667580 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 5/9] 10.222.0.62 () {56 vars in 1198 bytes} [Fri Sep 23 13:42:07 2022] GET /identity/v3/users/2ea3af238714467ab48e451a5210c19f => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:09.869858 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b7a525-b6ab-4b5e-b63e-ff128c0e763a None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:09.870419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b7a525-b6ab-4b5e-b63e-ff128c0e763a None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:09.870419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b7a525-b6ab-4b5e-b63e-ff128c0e763a None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:09.871753 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 5/10] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:09.878142 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec79f6f4-e879-4397-a7cc-d9cd8050e702 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:09.878565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec79f6f4-e879-4397-a7cc-d9cd8050e702 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:09.878799 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec79f6f4-e879-4397-a7cc-d9cd8050e702 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:09.899923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-ec79f6f4-e879-4397-a7cc-d9cd8050e702 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:09.901139 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-ec79f6f4-e879-4397-a7cc-d9cd8050e702 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:09.960145 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f86ec7-a0c6-4086-abaa-5e57a27d8905 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:09.960523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f86ec7-a0c6-4086-abaa-5e57a27d8905 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:09.960799 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f86ec7-a0c6-4086-abaa-5e57a27d8905 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:09.962118 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 6/11] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:42:09.967431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb24f652-1413-4e82-b948-3ff500098241 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:09.967710 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb24f652-1413-4e82-b948-3ff500098241 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:09.967985 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb24f652-1413-4e82-b948-3ff500098241 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:09.984509 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 6/12] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:10.009944 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG passlib.handlers.bcrypt [None req-eb24f652-1413-4e82-b948-3ff500098241 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81569) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 23 13:42:10.010354 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG passlib.handlers.bcrypt [None req-eb24f652-1413-4e82-b948-3ff500098241 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81569) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 23 13:42:10.059413 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-eb24f652-1413-4e82-b948-3ff500098241 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:10.061634 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-eb24f652-1413-4e82-b948-3ff500098241 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:10.161306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 7/13] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 23 13:42:10.255645 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec79f6f4-e879-4397-a7cc-d9cd8050e702 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:10.281127 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:10.358397 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec79f6f4-e879-4397-a7cc-d9cd8050e702 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:10.393357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aebe451-61d3-40b8-90d0-7d30e1aab843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:10.393989 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aebe451-61d3-40b8-90d0-7d30e1aab843 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:10.394272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aebe451-61d3-40b8-90d0-7d30e1aab843 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:10.394575 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aebe451-61d3-40b8-90d0-7d30e1aab843 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:10.406644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aebe451-61d3-40b8-90d0-7d30e1aab843 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:10.415222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-0aebe451-61d3-40b8-90d0-7d30e1aab843 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:42:10.415222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:10.415222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:10.415222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:10.415222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:10.415222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:10.415222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:10.415222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:10.415222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:10.415222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:10.415222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:10.415222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:10.415222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:10.415222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:10.415222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:10.415222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:10.416244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:10.416244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:10.416244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:10.416244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:10.416244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:10.416244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:10.416244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:10.416244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:10.416244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:10.416244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:10.416244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:10.416244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:10.416244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:10.416244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:10.416244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:10.416244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:10.417181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:10.417181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:10.417181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:10.417181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:10.417181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:10.417181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:10.417181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:10.417181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:10.417181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:10.417181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:10.417181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:42:10.417181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:42:10.417181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 7/14] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:42:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:10.425097 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb24f652-1413-4e82-b948-3ff500098241 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:10.427803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aebe451-61d3-40b8-90d0-7d30e1aab843 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:10.516129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:10.529456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb24f652-1413-4e82-b948-3ff500098241 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:10.531876 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:10.533702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aebe451-61d3-40b8-90d0-7d30e1aab843 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:10.568275 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3f7329-855d-4370-bf1a-8cbc523c4128 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:10.568789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3f293b-f294-4a1d-b623-a4ff807f8503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:10.569140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3f7329-855d-4370-bf1a-8cbc523c4128 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:10.569425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3f293b-f294-4a1d-b623-a4ff807f8503 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:10.569511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3f7329-855d-4370-bf1a-8cbc523c4128 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:10.569708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3f7329-855d-4370-bf1a-8cbc523c4128 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:10.569790 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3f293b-f294-4a1d-b623-a4ff807f8503 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:10.569980 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3f293b-f294-4a1d-b623-a4ff807f8503 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:10.570589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3f7329-855d-4370-bf1a-8cbc523c4128 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:10.570897 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3f7329-855d-4370-bf1a-8cbc523c4128 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:10.572322 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3f7329-855d-4370-bf1a-8cbc523c4128 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:10.585239 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 8/15] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:10.590376 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3f7329-855d-4370-bf1a-8cbc523c4128 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:10.640119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3f293b-f294-4a1d-b623-a4ff807f8503 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:10.641606 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3f293b-f294-4a1d-b623-a4ff807f8503 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:10.690504 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3f7329-855d-4370-bf1a-8cbc523c4128 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:10.721257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:10.723636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17880149-9a19-42bf-8511-eb44b55cff16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:10.724490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17880149-9a19-42bf-8511-eb44b55cff16 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:10.724770 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17880149-9a19-42bf-8511-eb44b55cff16 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:10.725042 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17880149-9a19-42bf-8511-eb44b55cff16 None admin] PATH_INFO: `/v3/users/2ea3af238714467ab48e451a5210c19f` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:10.726657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17880149-9a19-42bf-8511-eb44b55cff16 None admin] RBAC: Authorizing `identity:get_user(user_id=2ea3af238714467ab48e451a5210c19f)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:10.728055 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17880149-9a19-42bf-8511-eb44b55cff16 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:10.734449 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 9/16] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:10 2022] GET /identity/v3/users/2ea3af238714467ab48e451a5210c19f => generated 247 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:10.735831 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17880149-9a19-42bf-8511-eb44b55cff16 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:10.740344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 8/17] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:42:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 23 13:42:10.745770 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3f293b-f294-4a1d-b623-a4ff807f8503 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:10.770036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:10.835660 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17880149-9a19-42bf-8511-eb44b55cff16 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:10.841283 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3f293b-f294-4a1d-b623-a4ff807f8503 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:10.872228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39aeb3b5-910c-47bd-82fc-85358318a15d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:10.872978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aeb3b5-910c-47bd-82fc-85358318a15d None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:10.873651 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aeb3b5-910c-47bd-82fc-85358318a15d None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:10.873925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39aeb3b5-910c-47bd-82fc-85358318a15d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:10.875109 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39aeb3b5-910c-47bd-82fc-85358318a15d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:10.876090 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3d41ea-8c07-4761-a0e5-3d935df67fc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:10.876806 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39aeb3b5-910c-47bd-82fc-85358318a15d None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:10.877079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3d41ea-8c07-4761-a0e5-3d935df67fc2 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:10.877416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3d41ea-8c07-4761-a0e5-3d935df67fc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:10.877740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f3d41ea-8c07-4761-a0e5-3d935df67fc2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:10.878545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 10/18] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:42:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:10.884084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39aeb3b5-910c-47bd-82fc-85358318a15d None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:10.888827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3d41ea-8c07-4761-a0e5-3d935df67fc2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:10.890297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f3d41ea-8c07-4761-a0e5-3d935df67fc2 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:10.907148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-4f3d41ea-8c07-4761-a0e5-3d935df67fc2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:42:10.907148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:10.907148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:10.907148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:10.907148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:10.907148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:10.907148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:10.907148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:10.907148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:10.907148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:10.907148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:10.907148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:10.907148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:42:10.907148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:42:10.907148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:42:10.907148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:42:10.908123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:10.908123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:10.908123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:10.908123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:10.908123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:10.908123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:10.908123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:10.908123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:10.908123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:10.908123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:10.908123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:10.908123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:10.908123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:10.908123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:10.908123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:10.908123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:10.908123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:42:10.908902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:42:10.908902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:42:10.908902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:42:10.908902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 9/19] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:42:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:42:10.913998 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3d41ea-8c07-4761-a0e5-3d935df67fc2 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:10.954017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:10.988645 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39aeb3b5-910c-47bd-82fc-85358318a15d None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:10.990246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:11.015798 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f3d41ea-8c07-4761-a0e5-3d935df67fc2 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:11.022644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b2fd41-d907-40c9-8bfd-272ff95acfa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:11.023257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b2fd41-d907-40c9-8bfd-272ff95acfa1 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:11.023534 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b2fd41-d907-40c9-8bfd-272ff95acfa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:11.023801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b2fd41-d907-40c9-8bfd-272ff95acfa1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:11.024563 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b2fd41-d907-40c9-8bfd-272ff95acfa1 None admin] RBAC: Adding query filter params (role.id=727f0fa598e54b9a929d2cf6277c875e, scope.domain.id=default, user.id=2ea3af238714467ab48e451a5210c19f) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:11.024878 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b2fd41-d907-40c9-8bfd-272ff95acfa1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:11.026351 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b2fd41-d907-40c9-8bfd-272ff95acfa1 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:11.035558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 11/20] 10.222.0.62 () {56 vars in 1322 bytes} [Fri Sep 23 13:42:10 2022] GET /identity/v3/role_assignments?user.id=2ea3af238714467ab48e451a5210c19f&scope.domain.id=default&role.id=727f0fa598e54b9a929d2cf6277c875e => generated 233 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:11.051334 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9d5609-facc-4490-954d-bff34024398f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:11.051967 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9d5609-facc-4490-954d-bff34024398f None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:11.052245 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9d5609-facc-4490-954d-bff34024398f None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:11.052522 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9d5609-facc-4490-954d-bff34024398f None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:11.053192 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9d5609-facc-4490-954d-bff34024398f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:11.053492 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9d5609-facc-4490-954d-bff34024398f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:11.055069 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9d5609-facc-4490-954d-bff34024398f None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:11.075124 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 10/21] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:42:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:13.703189 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2a030c-05e0-4adf-b2ea-3bfa6c7dbe1a None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:13.704215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2a030c-05e0-4adf-b2ea-3bfa6c7dbe1a None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:13.704730 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2a030c-05e0-4adf-b2ea-3bfa6c7dbe1a None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:13.706946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 12/22] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:13 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:13.712972 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c8e583-1bd2-4b70-85f9-51e5dbb21260 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:13.713951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c8e583-1bd2-4b70-85f9-51e5dbb21260 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:13.714502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c8e583-1bd2-4b70-85f9-51e5dbb21260 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:13.739251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-d6c8e583-1bd2-4b70-85f9-51e5dbb21260 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:13.740965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d6c8e583-1bd2-4b70-85f9-51e5dbb21260 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:13.777691 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3326221c-3a11-430b-9dbe-cc454fc78493 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:13.778316 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3326221c-3a11-430b-9dbe-cc454fc78493 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:13.778828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3326221c-3a11-430b-9dbe-cc454fc78493 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:13.780727 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 13/23] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:13.785548 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2167c7-2cc8-4925-bdb7-73bb6504c6a8 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:13.786110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2167c7-2cc8-4925-bdb7-73bb6504c6a8 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:13.786637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b2167c7-2cc8-4925-bdb7-73bb6504c6a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:13.819380 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-9b2167c7-2cc8-4925-bdb7-73bb6504c6a8 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:13.820998 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9b2167c7-2cc8-4925-bdb7-73bb6504c6a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:13.867735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 11/24] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:13.937711 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 14/25] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:14.036408 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab11d16-b12d-43b2-8d64-b0c52c2a1f4e None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:14.036691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab11d16-b12d-43b2-8d64-b0c52c2a1f4e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:14.036887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab11d16-b12d-43b2-8d64-b0c52c2a1f4e None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:14.038517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 12/26] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:14.059247 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07456b7f-20c0-4d39-88b8-c2cdb716e76f None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:14.059550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07456b7f-20c0-4d39-88b8-c2cdb716e76f None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:14.059840 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07456b7f-20c0-4d39-88b8-c2cdb716e76f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:14.087329 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-07456b7f-20c0-4d39-88b8-c2cdb716e76f None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:14.088527 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-07456b7f-20c0-4d39-88b8-c2cdb716e76f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:14.188695 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 15/27] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:14.255039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b9ccc8-2deb-42b7-b3b6-0baa1644e9a9 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:14.255356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b9ccc8-2deb-42b7-b3b6-0baa1644e9a9 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:14.255602 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b9ccc8-2deb-42b7-b3b6-0baa1644e9a9 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:14.257292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 13/28] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:14.268258 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53027145-5a3d-4738-ac6a-106e8d72362f None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:14.268844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53027145-5a3d-4738-ac6a-106e8d72362f None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:14.269328 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53027145-5a3d-4738-ac6a-106e8d72362f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:14.299422 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-53027145-5a3d-4738-ac6a-106e8d72362f None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:14.300937 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-53027145-5a3d-4738-ac6a-106e8d72362f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:14.320535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b9ccc8-2deb-42b7-b3b6-0baa1644e9a9 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:14.334478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:14.375094 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:14.418079 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 16/29] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:14.455640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53027145-5a3d-4738-ac6a-106e8d72362f None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:14.548219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b9ccc8-2deb-42b7-b3b6-0baa1644e9a9 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:14.609944 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723fb62b-f099-4c6b-8b09-25062341a1d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:14.610517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723fb62b-f099-4c6b-8b09-25062341a1d9 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:14.610796 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723fb62b-f099-4c6b-8b09-25062341a1d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:14.611064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723fb62b-f099-4c6b-8b09-25062341a1d9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:14.632097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723fb62b-f099-4c6b-8b09-25062341a1d9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:14.633571 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-723fb62b-f099-4c6b-8b09-25062341a1d9 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:14.649295 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53027145-5a3d-4738-ac6a-106e8d72362f None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:14.664747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-723fb62b-f099-4c6b-8b09-25062341a1d9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:42:14.664747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:14.664747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:14.664747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:14.664747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:14.664747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:14.664747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:14.664747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:14.664747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:14.664747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:14.664747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:14.664747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:14.664747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:42:14.664747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:42:14.664747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:42:14.664747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:42:14.665705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:14.665705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:14.665705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:14.665705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:14.665705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:14.665705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:14.665705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:14.665705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:14.665705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:14.665705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:14.665705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:14.665705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:14.665705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:14.665705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:14.665705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:14.665705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:14.665705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:42:14.666528 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:42:14.666528 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:42:14.666528 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:42:14.668088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 14/30] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:42:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 348 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:14.668870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723fb62b-f099-4c6b-8b09-25062341a1d9 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:14.714045 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cb92f6-9826-47fc-92a8-3e5032d6ff9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:14.714965 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cb92f6-9826-47fc-92a8-3e5032d6ff9a None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:14.716256 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cb92f6-9826-47fc-92a8-3e5032d6ff9a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:14.719347 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cb92f6-9826-47fc-92a8-3e5032d6ff9a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:14.740849 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59cb92f6-9826-47fc-92a8-3e5032d6ff9a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:14.761148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-59cb92f6-9826-47fc-92a8-3e5032d6ff9a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:42:14.761148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:14.761148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:14.761148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:14.761148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:14.761148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:14.761148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:14.761148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:14.761148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:14.761148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:14.761148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:14.761148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:14.761148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:14.761148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:14.761148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:14.761148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:14.762228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:14.762228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:14.762228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:14.762228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:14.762228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:14.762228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:14.762228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:14.762228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:14.762228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:14.762228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:14.762228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:14.762228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:14.762228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:14.762228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:14.762228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:14.762228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:14.763731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:14.763731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:14.763731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:14.763731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:14.763731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:14.763731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:14.763731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:14.763731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:14.763731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:14.763731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:14.763731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:42:14.763731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:42:14.766919 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 17/31] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:42:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:14.768976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cb92f6-9826-47fc-92a8-3e5032d6ff9a None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:14.809607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:14.830966 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:14.862417 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-723fb62b-f099-4c6b-8b09-25062341a1d9 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:14.911731 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee622b33-0136-4bb0-a701-4d5e4501cb29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:14.912464 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee622b33-0136-4bb0-a701-4d5e4501cb29 None admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:14.912744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee622b33-0136-4bb0-a701-4d5e4501cb29 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:14.913016 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee622b33-0136-4bb0-a701-4d5e4501cb29 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:14.913626 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cb92f6-9826-47fc-92a8-3e5032d6ff9a None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:14.913728 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee622b33-0136-4bb0-a701-4d5e4501cb29 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:14.915326 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee622b33-0136-4bb0-a701-4d5e4501cb29 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:14.951727 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99681d13-832f-4726-add8-dabcee0e0309 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:14.952455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99681d13-832f-4726-add8-dabcee0e0309 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:14.952737 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99681d13-832f-4726-add8-dabcee0e0309 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:14.953008 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99681d13-832f-4726-add8-dabcee0e0309 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:14.953657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99681d13-832f-4726-add8-dabcee0e0309 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:14.953957 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99681d13-832f-4726-add8-dabcee0e0309 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:14.955375 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99681d13-832f-4726-add8-dabcee0e0309 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:14.977524 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 18/32] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:42:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:14.978501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99681d13-832f-4726-add8-dabcee0e0309 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:15.064352 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:15.087604 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99681d13-832f-4726-add8-dabcee0e0309 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:15.132946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe46bfb5-1026-4cc9-b8ce-16b22025aa91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:15.133490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe46bfb5-1026-4cc9-b8ce-16b22025aa91 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:15.133774 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe46bfb5-1026-4cc9-b8ce-16b22025aa91 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:15.134047 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe46bfb5-1026-4cc9-b8ce-16b22025aa91 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:15.134783 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe46bfb5-1026-4cc9-b8ce-16b22025aa91 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:15.135089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe46bfb5-1026-4cc9-b8ce-16b22025aa91 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:15.136559 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe46bfb5-1026-4cc9-b8ce-16b22025aa91 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:15.146446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 19/33] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:15.147279 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe46bfb5-1026-4cc9-b8ce-16b22025aa91 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:15.155933 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 15/34] 10.222.0.62 () {60 vars in 1131 bytes} [Fri Sep 23 13:42:14 2022] POST /identity/v3/roles => generated 219 bytes in 487 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:42:15.156951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee622b33-0136-4bb0-a701-4d5e4501cb29 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:15.272012 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe46bfb5-1026-4cc9-b8ce-16b22025aa91 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:15.315419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee622b33-0136-4bb0-a701-4d5e4501cb29 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:15.327558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbe93ba-e48f-4a86-b4af-60c90d72194c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:15.328703 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbe93ba-e48f-4a86-b4af-60c90d72194c None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:15.329063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbe93ba-e48f-4a86-b4af-60c90d72194c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:15.329416 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbe93ba-e48f-4a86-b4af-60c90d72194c None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:15.330520 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbe93ba-e48f-4a86-b4af-60c90d72194c None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:15.332337 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbe93ba-e48f-4a86-b4af-60c90d72194c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:15.357741 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c14f87-86bb-42bf-9e2a-2e93bf87c8a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:15.358412 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c14f87-86bb-42bf-9e2a-2e93bf87c8a0 None admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:15.358695 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c14f87-86bb-42bf-9e2a-2e93bf87c8a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:15.358969 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c14f87-86bb-42bf-9e2a-2e93bf87c8a0 None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:15.359632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c14f87-86bb-42bf-9e2a-2e93bf87c8a0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:15.361293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49c14f87-86bb-42bf-9e2a-2e93bf87c8a0 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:15.367707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:15.397900 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:15.478243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 16/35] 10.222.0.62 () {60 vars in 1141 bytes} [Fri Sep 23 13:42:15 2022] POST /identity/v3/projects => generated 301 bytes in 322 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:42:15.479133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c14f87-86bb-42bf-9e2a-2e93bf87c8a0 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:15.516289 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 20/36] 10.222.0.62 () {60 vars in 1131 bytes} [Fri Sep 23 13:42:15 2022] POST /identity/v3/roles => generated 225 bytes in 363 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:42:15.516289 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c07ed23-144f-451b-8ae7-5a2b474d2a49 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:15.516289 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c07ed23-144f-451b-8ae7-5a2b474d2a49 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:15.516289 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c07ed23-144f-451b-8ae7-5a2b474d2a49 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:15.516289 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 21/37] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:15.516289 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98815a0c-0af1-420d-bc73-2e1f0f6d4e8f None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:15.516289 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98815a0c-0af1-420d-bc73-2e1f0f6d4e8f None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:15.516837 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98815a0c-0af1-420d-bc73-2e1f0f6d4e8f None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:15.519118 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 22/38] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:15.525200 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca51b3f8-2b09-44c2-9386-99c361462296 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:15.525483 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca51b3f8-2b09-44c2-9386-99c361462296 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:15.525796 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca51b3f8-2b09-44c2-9386-99c361462296 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:15.553821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-ca51b3f8-2b09-44c2-9386-99c361462296 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:15.555594 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ca51b3f8-2b09-44c2-9386-99c361462296 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:15.645074 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 23/39] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:15.645074 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef40e7d3-7ed7-47b5-86f4-bf793b5f91d0 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:15.645074 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef40e7d3-7ed7-47b5-86f4-bf793b5f91d0 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:15.645074 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef40e7d3-7ed7-47b5-86f4-bf793b5f91d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:15.653607 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c14f87-86bb-42bf-9e2a-2e93bf87c8a0 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:15.676094 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-ef40e7d3-7ed7-47b5-86f4-bf793b5f91d0 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:15.677331 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ef40e7d3-7ed7-47b5-86f4-bf793b5f91d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:15.740329 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10c5e70-4fd1-4a9e-a0be-d96d55c9db80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:15.740987 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10c5e70-4fd1-4a9e-a0be-d96d55c9db80 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:15.741270 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10c5e70-4fd1-4a9e-a0be-d96d55c9db80 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:15.741548 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10c5e70-4fd1-4a9e-a0be-d96d55c9db80 None admin] PATH_INFO: `/v3/users/2ea3af238714467ab48e451a5210c19f` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:15.742890 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10c5e70-4fd1-4a9e-a0be-d96d55c9db80 None admin] RBAC: Authorizing `identity:get_user(user_id=2ea3af238714467ab48e451a5210c19f)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:15.744362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c10c5e70-4fd1-4a9e-a0be-d96d55c9db80 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:15.745950 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 17/40] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:15 2022] GET /identity/v3/users/2ea3af238714467ab48e451a5210c19f => generated 247 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:15.754895 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10c5e70-4fd1-4a9e-a0be-d96d55c9db80 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:15.762164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 24/41] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:15.823197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:15.867763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10c5e70-4fd1-4a9e-a0be-d96d55c9db80 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:15.902270 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744b9682-fa6a-41e5-978e-a143a8326230 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:15.902976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744b9682-fa6a-41e5-978e-a143a8326230 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:15.903265 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744b9682-fa6a-41e5-978e-a143a8326230 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:15.903540 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744b9682-fa6a-41e5-978e-a143a8326230 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:15.905039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744b9682-fa6a-41e5-978e-a143a8326230 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:15.907466 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744b9682-fa6a-41e5-978e-a143a8326230 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:15.909247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 18/42] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:42:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:15.914697 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef40e7d3-7ed7-47b5-86f4-bf793b5f91d0 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:15.949886 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744b9682-fa6a-41e5-978e-a143a8326230 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:15.978985 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:16.024034 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef40e7d3-7ed7-47b5-86f4-bf793b5f91d0 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:16.059846 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38f9943-9154-49ac-8d18-cc262752be37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:16.061101 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38f9943-9154-49ac-8d18-cc262752be37 None admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:16.061387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38f9943-9154-49ac-8d18-cc262752be37 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:16.061669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38f9943-9154-49ac-8d18-cc262752be37 None admin] PATH_INFO: `/v3/domains/default/users/2ea3af238714467ab48e451a5210c19f/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:16.063766 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38f9943-9154-49ac-8d18-cc262752be37 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=2ea3af238714467ab48e451a5210c19f, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:16.065403 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38f9943-9154-49ac-8d18-cc262752be37 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:16.146000 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 25/43] 10.222.0.62 () {58 vars in 1361 bytes} [Fri Sep 23 13:42:15 2022] PUT /identity/v3/domains/default/users/2ea3af238714467ab48e451a5210c19f/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 13:42:16.146846 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38f9943-9154-49ac-8d18-cc262752be37 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:16.179335 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744b9682-fa6a-41e5-978e-a143a8326230 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:16.207319 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:16.222248 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95193145-322a-4aa0-8929-7498a2bda9d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:16.223287 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95193145-322a-4aa0-8929-7498a2bda9d0 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:16.223750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95193145-322a-4aa0-8929-7498a2bda9d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:16.224209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95193145-322a-4aa0-8929-7498a2bda9d0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:16.225817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95193145-322a-4aa0-8929-7498a2bda9d0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:16.227827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95193145-322a-4aa0-8929-7498a2bda9d0 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:16.229844 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 19/44] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:42:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:16.235717 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95193145-322a-4aa0-8929-7498a2bda9d0 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:16.280356 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38f9943-9154-49ac-8d18-cc262752be37 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:16.300877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:16.325018 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437cf045-fde0-40c1-b83c-22119d1ce7f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:16.325702 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437cf045-fde0-40c1-b83c-22119d1ce7f5 None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:16.325988 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437cf045-fde0-40c1-b83c-22119d1ce7f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:16.326269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437cf045-fde0-40c1-b83c-22119d1ce7f5 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:16.327556 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437cf045-fde0-40c1-b83c-22119d1ce7f5 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:16.328874 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437cf045-fde0-40c1-b83c-22119d1ce7f5 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:16.355961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95193145-322a-4aa0-8929-7498a2bda9d0 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:16.391310 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e4669d-9df4-4381-a489-c75d8d0c85d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:16.391972 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e4669d-9df4-4381-a489-c75d8d0c85d1 None admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:16.392251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e4669d-9df4-4381-a489-c75d8d0c85d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:16.392524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e4669d-9df4-4381-a489-c75d8d0c85d1 None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:16.393170 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e4669d-9df4-4381-a489-c75d8d0c85d1 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:16.394628 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e4669d-9df4-4381-a489-c75d8d0c85d1 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:16.403545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 26/45] 10.222.0.62 () {60 vars in 1131 bytes} [Fri Sep 23 13:42:16 2022] POST /identity/v3/roles => generated 223 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:42:16.518143 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 20/46] 10.222.0.62 () {60 vars in 1141 bytes} [Fri Sep 23 13:42:16 2022] POST /identity/v3/projects => generated 312 bytes in 283 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 13:42:18.327849 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a69e28-bb82-42e8-80d3-83d8a747269c None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:18.328384 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a69e28-bb82-42e8-80d3-83d8a747269c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:18.328384 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a69e28-bb82-42e8-80d3-83d8a747269c None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:18.329780 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 27/47] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:18.335178 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53bdaef-c4f0-430c-b001-34eec842c765 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:18.335592 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53bdaef-c4f0-430c-b001-34eec842c765 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:18.335727 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53bdaef-c4f0-430c-b001-34eec842c765 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:18.354223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-a53bdaef-c4f0-430c-b001-34eec842c765 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:18.355775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a53bdaef-c4f0-430c-b001-34eec842c765 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:18.395176 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:18.421425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 21/48] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:18.688688 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a69e28-bb82-42e8-80d3-83d8a747269c None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:18.699361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c56cfd1-f48d-456c-aba8-73610184497d None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:18.699677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c56cfd1-f48d-456c-aba8-73610184497d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:18.699964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c56cfd1-f48d-456c-aba8-73610184497d None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:18.701506 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 22/49] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:42:18.706522 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df9a1c4-ebce-4bee-84a8-896b4d30959c None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:18.706808 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df9a1c4-ebce-4bee-84a8-896b4d30959c None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:18.707085 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df9a1c4-ebce-4bee-84a8-896b4d30959c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:18.724649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-9df9a1c4-ebce-4bee-84a8-896b4d30959c None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:18.725843 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-9df9a1c4-ebce-4bee-84a8-896b4d30959c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:18.792778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 23/50] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 23 13:42:18.793230 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a69e28-bb82-42e8-80d3-83d8a747269c None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:18.827752 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a9dcc6-77d3-4fda-8084-2a664d1b3c7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:18.828380 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a9dcc6-77d3-4fda-8084-2a664d1b3c7f None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:18.828665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a9dcc6-77d3-4fda-8084-2a664d1b3c7f None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:18.828947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a9dcc6-77d3-4fda-8084-2a664d1b3c7f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:18.839306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a9dcc6-77d3-4fda-8084-2a664d1b3c7f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:18.844333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-e2a9dcc6-77d3-4fda-8084-2a664d1b3c7f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:42:18.844333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:18.844333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:18.844333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:18.844333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:18.844333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:18.844333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:18.844333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:18.844333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:18.844333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:18.844333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:18.844333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:18.844333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:18.844333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:18.844333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:18.844333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:18.845319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:18.845319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:18.845319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:18.845319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:18.845319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:18.845319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:18.845319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:18.845319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:18.845319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:18.845319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:18.845319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:18.845319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:18.845319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:18.845319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:18.845319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:18.845319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:18.846148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:18.846148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:18.846148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:18.846148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:18.846148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:18.846148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:18.846148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:18.846148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:18.846148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:18.846148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:18.846148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:42:18.846148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:42:18.846148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 28/51] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:42:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:18.851209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df9a1c4-ebce-4bee-84a8-896b4d30959c None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:18.948073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df9a1c4-ebce-4bee-84a8-896b4d30959c None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:18.950230 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:18.980893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f221969-8c9b-4db8-bb79-dcf4f2fe83ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:18.981502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f221969-8c9b-4db8-bb79-dcf4f2fe83ad None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:18.982512 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f221969-8c9b-4db8-bb79-dcf4f2fe83ad None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:18.982512 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f221969-8c9b-4db8-bb79-dcf4f2fe83ad None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:18.983476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f221969-8c9b-4db8-bb79-dcf4f2fe83ad None admin] RBAC: Adding query filter params (name=admin) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:18.983778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f221969-8c9b-4db8-bb79-dcf4f2fe83ad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:18.985154 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f221969-8c9b-4db8-bb79-dcf4f2fe83ad None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:18.997507 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 24/52] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:19.003390 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a9dcc6-77d3-4fda-8084-2a664d1b3c7f None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:19.042786 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:19.085299 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f221969-8c9b-4db8-bb79-dcf4f2fe83ad None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:19.107580 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a9dcc6-77d3-4fda-8084-2a664d1b3c7f None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:19.144178 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3c6064-9d38-45dd-9cfc-a9f74d3b37c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:19.144820 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3c6064-9d38-45dd-9cfc-a9f74d3b37c0 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:19.145082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3c6064-9d38-45dd-9cfc-a9f74d3b37c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:19.145359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3c6064-9d38-45dd-9cfc-a9f74d3b37c0 None admin] PATH_INFO: `/v3/users/2ea3af238714467ab48e451a5210c19f` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:19.146504 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3c6064-9d38-45dd-9cfc-a9f74d3b37c0 None admin] RBAC: Authorizing `identity:get_user(user_id=2ea3af238714467ab48e451a5210c19f)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:19.147944 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3c6064-9d38-45dd-9cfc-a9f74d3b37c0 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:19.149732 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 29/53] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:19 2022] GET /identity/v3/users/2ea3af238714467ab48e451a5210c19f => generated 247 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:19.155803 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3c6064-9d38-45dd-9cfc-a9f74d3b37c0 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:19.190711 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f221969-8c9b-4db8-bb79-dcf4f2fe83ad None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:19.228151 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ccd5662-239a-492f-aebb-e8510f3d8300 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:19.228779 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccd5662-239a-492f-aebb-e8510f3d8300 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:19.229060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccd5662-239a-492f-aebb-e8510f3d8300 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:19.229328 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ccd5662-239a-492f-aebb-e8510f3d8300 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:19.230460 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ccd5662-239a-492f-aebb-e8510f3d8300 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:19.232495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ccd5662-239a-492f-aebb-e8510f3d8300 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:19.234143 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 25/54] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:42:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:19.239773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ccd5662-239a-492f-aebb-e8510f3d8300 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:19.261575 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3c6064-9d38-45dd-9cfc-a9f74d3b37c0 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:19.274347 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:19.351067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ccd5662-239a-492f-aebb-e8510f3d8300 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:19.379415 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:19.387541 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f0d879-b3be-4951-bc9d-3ba08d038101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:19.391602 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f0d879-b3be-4951-bc9d-3ba08d038101 None admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:19.391602 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f0d879-b3be-4951-bc9d-3ba08d038101 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:19.391602 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f0d879-b3be-4951-bc9d-3ba08d038101 None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:19.391602 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f0d879-b3be-4951-bc9d-3ba08d038101 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:19.391602 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f0d879-b3be-4951-bc9d-3ba08d038101 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:19.408355 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6d7378-9be1-41e0-97d1-31209729831c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:19.409326 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6d7378-9be1-41e0-97d1-31209729831c None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:19.409606 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6d7378-9be1-41e0-97d1-31209729831c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:19.409875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6d7378-9be1-41e0-97d1-31209729831c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:19.411004 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6d7378-9be1-41e0-97d1-31209729831c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:19.412455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6d7378-9be1-41e0-97d1-31209729831c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:19.414005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 30/55] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:42:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:19.420276 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6d7378-9be1-41e0-97d1-31209729831c None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:19.477937 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 26/56] 10.222.0.62 () {60 vars in 1140 bytes} [Fri Sep 23 13:42:19 2022] POST /identity/v3/projects => generated 298 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 13:42:19.523097 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6d7378-9be1-41e0-97d1-31209729831c None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:19.559265 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb5d7f9-dddb-407e-af84-a17a4d2aec58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:19.560489 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb5d7f9-dddb-407e-af84-a17a4d2aec58 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:19.560772 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb5d7f9-dddb-407e-af84-a17a4d2aec58 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:19.561042 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb5d7f9-dddb-407e-af84-a17a4d2aec58 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:19.561782 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb5d7f9-dddb-407e-af84-a17a4d2aec58 None admin] RBAC: Adding query filter params (role.id=727f0fa598e54b9a929d2cf6277c875e, scope.domain.id=default, user.id=2ea3af238714467ab48e451a5210c19f) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:19.562087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb5d7f9-dddb-407e-af84-a17a4d2aec58 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:19.563679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb5d7f9-dddb-407e-af84-a17a4d2aec58 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:19.570257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 31/57] 10.222.0.62 () {56 vars in 1322 bytes} [Fri Sep 23 13:42:19 2022] GET /identity/v3/role_assignments?user.id=2ea3af238714467ab48e451a5210c19f&scope.domain.id=default&role.id=727f0fa598e54b9a929d2cf6277c875e => generated 530 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:21.727951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7beeeb6-d9ab-4cc0-823b-6b4bae0b9a10 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:21.728446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7beeeb6-d9ab-4cc0-823b-6b4bae0b9a10 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:21.728527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7beeeb6-d9ab-4cc0-823b-6b4bae0b9a10 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:21.729832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 27/58] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:21.735077 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e90203-5556-420b-80dc-0a37f68074c3 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:21.735482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e90203-5556-420b-80dc-0a37f68074c3 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:21.735627 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e90203-5556-420b-80dc-0a37f68074c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:21.753729 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-46e90203-5556-420b-80dc-0a37f68074c3 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:21.754931 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-46e90203-5556-420b-80dc-0a37f68074c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:21.784147 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:21.818723 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 32/59] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:22.088753 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7beeeb6-d9ab-4cc0-823b-6b4bae0b9a10 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:22.176258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:22.193470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7beeeb6-d9ab-4cc0-823b-6b4bae0b9a10 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:22.227857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6119dc-cded-4e1f-a7ea-f2a516b66dfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:22.228482 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6119dc-cded-4e1f-a7ea-f2a516b66dfe None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:22.228760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6119dc-cded-4e1f-a7ea-f2a516b66dfe None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:22.229036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6119dc-cded-4e1f-a7ea-f2a516b66dfe None admin] PATH_INFO: `/v3/domains/default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:22.230129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6119dc-cded-4e1f-a7ea-f2a516b66dfe None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:22.232268 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6119dc-cded-4e1f-a7ea-f2a516b66dfe None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:22.233953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 28/60] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:42:22 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:22.239585 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e90203-5556-420b-80dc-0a37f68074c3 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:22.334082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e90203-5556-420b-80dc-0a37f68074c3 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:22.366245 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83ddb9b-8984-4b6a-b76a-d8c52f049c4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:22.366814 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83ddb9b-8984-4b6a-b76a-d8c52f049c4e None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:22.367096 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83ddb9b-8984-4b6a-b76a-d8c52f049c4e None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:22.367369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83ddb9b-8984-4b6a-b76a-d8c52f049c4e None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:22.368024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83ddb9b-8984-4b6a-b76a-d8c52f049c4e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:22.369451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83ddb9b-8984-4b6a-b76a-d8c52f049c4e None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:22.481349 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 33/61] 10.222.0.62 () {60 vars in 1132 bytes} [Fri Sep 23 13:42:22 2022] POST /identity/v3/users => generated 275 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:42:25.799439 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f99910b-0954-46a0-9e0f-9c1d8550cf38 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:25.799981 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f99910b-0954-46a0-9e0f-9c1d8550cf38 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:25.799981 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f99910b-0954-46a0-9e0f-9c1d8550cf38 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:25.801399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 29/62] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:25.816678 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af1033e-b51f-47ea-bfb3-92cc27717f88 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:25.817173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af1033e-b51f-47ea-bfb3-92cc27717f88 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:25.817243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af1033e-b51f-47ea-bfb3-92cc27717f88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:25.839395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-3af1033e-b51f-47ea-bfb3-92cc27717f88 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:25.841144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3af1033e-b51f-47ea-bfb3-92cc27717f88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:25.863560 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:25.932384 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 34/63] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:25.941076 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea922696-4dba-46e1-9539-19bd71300ac0 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:25.941365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea922696-4dba-46e1-9539-19bd71300ac0 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:25.941646 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea922696-4dba-46e1-9539-19bd71300ac0 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:25.943039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 30/64] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:25.948378 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4620f15-c212-4fdc-9d91-0a4ec5e54c86 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:25.948677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4620f15-c212-4fdc-9d91-0a4ec5e54c86 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:25.948960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4620f15-c212-4fdc-9d91-0a4ec5e54c86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:25.975412 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-c4620f15-c212-4fdc-9d91-0a4ec5e54c86 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:25.976692 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-c4620f15-c212-4fdc-9d91-0a4ec5e54c86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:26.077186 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 35/65] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:26.126947 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d48a07-5bb2-4b7e-ac40-834c9d61dd47 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:26.127268 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d48a07-5bb2-4b7e-ac40-834c9d61dd47 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:26.127493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d48a07-5bb2-4b7e-ac40-834c9d61dd47 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:26.129119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 31/66] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:26.134392 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dd7872-bc5e-4e58-96ae-08bc42825017 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:26.134691 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dd7872-bc5e-4e58-96ae-08bc42825017 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:26.134971 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36dd7872-bc5e-4e58-96ae-08bc42825017 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:26.163691 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-36dd7872-bc5e-4e58-96ae-08bc42825017 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:26.164882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-36dd7872-bc5e-4e58-96ae-08bc42825017 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:26.246365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d48a07-5bb2-4b7e-ac40-834c9d61dd47 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:26.272824 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 36/67] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:26.370684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d48a07-5bb2-4b7e-ac40-834c9d61dd47 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:26.422253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd43ca9-a555-431f-9ecc-f6cd03d17f90 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:26.422584 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd43ca9-a555-431f-9ecc-f6cd03d17f90 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:26.422869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd43ca9-a555-431f-9ecc-f6cd03d17f90 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:26.423222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73c0a55-e033-4335-9472-e5509e7a814b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:26.424145 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73c0a55-e033-4335-9472-e5509e7a814b None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:26.424245 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 37/68] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:42:26.424661 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73c0a55-e033-4335-9472-e5509e7a814b None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:26.425122 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73c0a55-e033-4335-9472-e5509e7a814b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:26.437186 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d73c0a55-e033-4335-9472-e5509e7a814b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:26.438376 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecd121a-d179-432c-953b-b2e644ae2715 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:26.439080 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecd121a-d179-432c-953b-b2e644ae2715 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:26.439364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ecd121a-d179-432c-953b-b2e644ae2715 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:26.443537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-d73c0a55-e033-4335-9472-e5509e7a814b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:26.443537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:26.443537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:26.443537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:26.443537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:26.443537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:26.443537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:26.443537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:26.443537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:26.443537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:26.443537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:26.443537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:26.443537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:26.443537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:26.443537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:26.443537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:26.444363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:26.444363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:26.444363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:26.444363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:26.444363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:26.444363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:26.444363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:26.444363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:26.444363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:26.444363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:26.444363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:26.444363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:26.444363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:26.444363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:26.444363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:26.444363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:26.445113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:26.445113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:26.445113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:26.445113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:26.445113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:26.445113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:26.445113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:26.445113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:26.445113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:26.445113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:26.445113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:26.445113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:42:26.447349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 32/69] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:26.453862 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73c0a55-e033-4335-9472-e5509e7a814b None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:26.470700 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:26.479403 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-3ecd121a-d179-432c-953b-b2e644ae2715 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:26.480621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3ecd121a-d179-432c-953b-b2e644ae2715 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:26.499816 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:26.568645 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 38/70] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 23 13:42:26.569287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ecd121a-d179-432c-953b-b2e644ae2715 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:26.578382 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73c0a55-e033-4335-9472-e5509e7a814b None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:26.616349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd39147-cefb-42d6-8565-9ac96d3b4e54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:26.616990 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd39147-cefb-42d6-8565-9ac96d3b4e54 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:26.617271 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd39147-cefb-42d6-8565-9ac96d3b4e54 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:26.617545 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd39147-cefb-42d6-8565-9ac96d3b4e54 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:26.618218 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd39147-cefb-42d6-8565-9ac96d3b4e54 None admin] RBAC: Adding query filter params (name=member) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:26.618548 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd39147-cefb-42d6-8565-9ac96d3b4e54 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:26.620080 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd39147-cefb-42d6-8565-9ac96d3b4e54 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:26.629463 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 33/71] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:42:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:26.630767 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971a5a3e-b1b9-40e1-8155-565c7866764a None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:26.631145 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971a5a3e-b1b9-40e1-8155-565c7866764a None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:26.631425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971a5a3e-b1b9-40e1-8155-565c7866764a None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:26.632888 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 34/72] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:26.637868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971a5a3e-b1b9-40e1-8155-565c7866764a None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:26.686537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ecd121a-d179-432c-953b-b2e644ae2715 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:26.724531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cba45e2-3cb9-431b-973e-6832665ab7d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:26.725165 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cba45e2-3cb9-431b-973e-6832665ab7d3 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:26.725449 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cba45e2-3cb9-431b-973e-6832665ab7d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:26.725723 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cba45e2-3cb9-431b-973e-6832665ab7d3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:26.729945 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:26.737134 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cba45e2-3cb9-431b-973e-6832665ab7d3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:26.742075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-7cba45e2-3cb9-431b-973e-6832665ab7d3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:42:26.742075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:26.742075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:26.742075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:26.742075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:26.742075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:26.742075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:26.742075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:26.742075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:26.742075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:26.742075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:26.742075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:26.742075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:26.742075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:26.742075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:26.742075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:26.749443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:26.749443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:26.749443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:26.749443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:26.749443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:26.749443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:26.749443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:26.749443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:26.749443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:26.749443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:26.749443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:26.749443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:26.749443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:26.749443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:26.749443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:26.749443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:26.750209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:26.750209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:26.750209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:26.750209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:26.750209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:26.750209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:26.750209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:26.750209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:26.750209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:26.750209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:26.750209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:42:26.750209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:42:26.750209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 39/73] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:42:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:26.750209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae92adcf-54a2-4292-b732-14d07bfb2cb7 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:26.751157 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae92adcf-54a2-4292-b732-14d07bfb2cb7 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:26.751157 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae92adcf-54a2-4292-b732-14d07bfb2cb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:26.753650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:26.763194 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971a5a3e-b1b9-40e1-8155-565c7866764a None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:26.765019 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-ae92adcf-54a2-4292-b732-14d07bfb2cb7 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:26.766285 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ae92adcf-54a2-4292-b732-14d07bfb2cb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:26.799744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1eafea5-ec62-4e66-ab34-7b8b5c182b16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:26.800429 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eafea5-ec62-4e66-ab34-7b8b5c182b16 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:26.800710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eafea5-ec62-4e66-ab34-7b8b5c182b16 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:26.801034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eafea5-ec62-4e66-ab34-7b8b5c182b16 None admin] PATH_INFO: `/v3/users/15dbb8e9805348eb9d1237a19e8e15f2` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:26.817451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1eafea5-ec62-4e66-ab34-7b8b5c182b16 None admin] RBAC: Authorizing `identity:get_user(user_id=15dbb8e9805348eb9d1237a19e8e15f2)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:26.819377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1eafea5-ec62-4e66-ab34-7b8b5c182b16 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:26.826060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 35/74] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:26 2022] GET /identity/v3/users/15dbb8e9805348eb9d1237a19e8e15f2 => generated 275 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:26.826060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1eafea5-ec62-4e66-ab34-7b8b5c182b16 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:26.837491 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 40/75] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:26.839470 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae92adcf-54a2-4292-b732-14d07bfb2cb7 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:26.935446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1eafea5-ec62-4e66-ab34-7b8b5c182b16 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:26.950013 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae92adcf-54a2-4292-b732-14d07bfb2cb7 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:26.970589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91efb91a-61ed-4684-8d27-9d756d4e50a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:26.971560 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91efb91a-61ed-4684-8d27-9d756d4e50a5 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:26.971560 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91efb91a-61ed-4684-8d27-9d756d4e50a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:26.971822 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91efb91a-61ed-4684-8d27-9d756d4e50a5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:26.973716 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:26.982947 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91efb91a-61ed-4684-8d27-9d756d4e50a5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:26.987043 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdb50e4-9729-4ffc-a1d9-f19d0d25b8c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:26.987742 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdb50e4-9729-4ffc-a1d9-f19d0d25b8c9 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:26.987856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-91efb91a-61ed-4684-8d27-9d756d4e50a5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:26.987856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:26.987856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:26.987856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:26.987856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:26.987856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:26.987856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:26.987856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:26.987856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:26.987856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:26.987856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:26.987856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:26.987856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:26.987856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:26.987856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:26.987856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:26.988639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:26.988639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:26.988639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:26.988639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:26.988639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:26.988639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:26.988639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:26.988639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:26.988639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:26.988639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:26.988639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:26.988639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:26.988639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:26.988639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:26.988639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:26.988639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:26.989377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:26.989377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:26.989377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:26.989377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:26.989377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:26.989377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:26.989377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:26.989377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:26.989377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:26.989377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:26.989377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:26.989377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:42:26.990023 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdb50e4-9729-4ffc-a1d9-f19d0d25b8c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:26.990023 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdb50e4-9729-4ffc-a1d9-f19d0d25b8c9 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:26.990141 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 36/76] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:26.990204 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afdb50e4-9729-4ffc-a1d9-f19d0d25b8c9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:26.990204 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afdb50e4-9729-4ffc-a1d9-f19d0d25b8c9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:26.990318 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91efb91a-61ed-4684-8d27-9d756d4e50a5 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:26.991362 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afdb50e4-9729-4ffc-a1d9-f19d0d25b8c9 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:26.998956 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:27.002194 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 41/77] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:27.003049 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdb50e4-9729-4ffc-a1d9-f19d0d25b8c9 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:27.106875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdb50e4-9729-4ffc-a1d9-f19d0d25b8c9 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:27.107399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91efb91a-61ed-4684-8d27-9d756d4e50a5 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:27.142754 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd0947d-73e5-40b4-9763-18d2d0584c58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:27.143425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd0947d-73e5-40b4-9763-18d2d0584c58 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:27.143713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd0947d-73e5-40b4-9763-18d2d0584c58 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:27.143994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd0947d-73e5-40b4-9763-18d2d0584c58 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:27.145142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53da1f27-508b-4306-9346-1bdcdf863e9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:27.145771 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53da1f27-508b-4306-9346-1bdcdf863e9a None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:27.146051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53da1f27-508b-4306-9346-1bdcdf863e9a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:27.146321 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53da1f27-508b-4306-9346-1bdcdf863e9a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:27.157176 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53da1f27-508b-4306-9346-1bdcdf863e9a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:27.161924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-53da1f27-508b-4306-9346-1bdcdf863e9a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:42:27.161924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:27.161924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:27.161924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:27.161924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:27.161924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:27.161924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:27.161924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:27.161924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:27.161924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:27.161924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:27.161924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:27.161924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:27.161924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:27.161924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:27.161924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:27.162829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:27.162829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:27.162829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:27.162829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:27.162829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:27.162829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:27.162829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:27.162829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:27.162829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:27.162829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:27.162829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:27.162829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:27.162829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:27.162829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:27.162829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:27.162829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:27.163587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:27.163587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:27.163587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:27.163587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:27.163587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:27.163587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:27.163587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:27.163587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:27.163587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:27.163587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:27.163587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:42:27.163587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:42:27.163587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 42/78] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:42:27 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:27.164214 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53da1f27-508b-4306-9346-1bdcdf863e9a None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:27.208935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:27.216962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd0947d-73e5-40b4-9763-18d2d0584c58 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:27.218487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd0947d-73e5-40b4-9763-18d2d0584c58 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:27.219776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:27.261514 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53da1f27-508b-4306-9346-1bdcdf863e9a None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:27.296010 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26de271d-67d5-4514-b674-f887613b2532 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:27.296613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26de271d-67d5-4514-b674-f887613b2532 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:27.296897 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26de271d-67d5-4514-b674-f887613b2532 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:27.297171 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26de271d-67d5-4514-b674-f887613b2532 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:27.297837 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26de271d-67d5-4514-b674-f887613b2532 None admin] RBAC: Adding query filter params (name=member) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:27.298136 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26de271d-67d5-4514-b674-f887613b2532 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:27.299544 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26de271d-67d5-4514-b674-f887613b2532 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:27.307657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 43/79] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:42:27 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:27.308247 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26de271d-67d5-4514-b674-f887613b2532 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:27.326925 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 37/80] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:42:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:42:27.326925 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd0947d-73e5-40b4-9763-18d2d0584c58 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:27.413073 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26de271d-67d5-4514-b674-f887613b2532 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:27.421073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:27.427767 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd0947d-73e5-40b4-9763-18d2d0584c58 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:27.430289 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:27.447485 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba061261-d3ef-4645-b75b-55216fcb8d2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:27.448114 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba061261-d3ef-4645-b75b-55216fcb8d2a None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:27.448395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba061261-d3ef-4645-b75b-55216fcb8d2a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:27.448669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba061261-d3ef-4645-b75b-55216fcb8d2a None admin] PATH_INFO: `/v3/users/2ea3af238714467ab48e451a5210c19f` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:27.449851 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba061261-d3ef-4645-b75b-55216fcb8d2a None admin] RBAC: Authorizing `identity:get_user(user_id=2ea3af238714467ab48e451a5210c19f)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:27.451822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba061261-d3ef-4645-b75b-55216fcb8d2a None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:27.453482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 44/81] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:27 2022] GET /identity/v3/users/2ea3af238714467ab48e451a5210c19f => generated 247 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:27.454243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba061261-d3ef-4645-b75b-55216fcb8d2a None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:27.469257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688999d7-c774-429a-b1f8-d92029495776 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:27.469257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688999d7-c774-429a-b1f8-d92029495776 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:27.469257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688999d7-c774-429a-b1f8-d92029495776 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:27.469257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688999d7-c774-429a-b1f8-d92029495776 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:27.469869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688999d7-c774-429a-b1f8-d92029495776 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:27.471431 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688999d7-c774-429a-b1f8-d92029495776 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:27.473188 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 38/82] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:42:27 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:27.473782 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688999d7-c774-429a-b1f8-d92029495776 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:27.553909 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba061261-d3ef-4645-b75b-55216fcb8d2a None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:27.578564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688999d7-c774-429a-b1f8-d92029495776 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:27.588104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f142d436-b5a6-43ea-b85e-ef3cff0f6e00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:27.588717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f142d436-b5a6-43ea-b85e-ef3cff0f6e00 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:27.589154 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f142d436-b5a6-43ea-b85e-ef3cff0f6e00 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:27.589439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f142d436-b5a6-43ea-b85e-ef3cff0f6e00 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:27.590111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f142d436-b5a6-43ea-b85e-ef3cff0f6e00 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:27.590594 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f142d436-b5a6-43ea-b85e-ef3cff0f6e00 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:27.592625 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f142d436-b5a6-43ea-b85e-ef3cff0f6e00 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:27.600501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 45/83] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:42:27 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:27.601440 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f142d436-b5a6-43ea-b85e-ef3cff0f6e00 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:27.614534 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a936481f-65fc-4436-bc1b-1d7aa33c23c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:27.615163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a936481f-65fc-4436-bc1b-1d7aa33c23c6 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:27.615443 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a936481f-65fc-4436-bc1b-1d7aa33c23c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:27.615715 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a936481f-65fc-4436-bc1b-1d7aa33c23c6 None admin] PATH_INFO: `/v3/users/15dbb8e9805348eb9d1237a19e8e15f2` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:27.616856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a936481f-65fc-4436-bc1b-1d7aa33c23c6 None admin] RBAC: Authorizing `identity:get_user(user_id=15dbb8e9805348eb9d1237a19e8e15f2)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:27.618289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a936481f-65fc-4436-bc1b-1d7aa33c23c6 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:27.619873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 39/84] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:27 2022] GET /identity/v3/users/15dbb8e9805348eb9d1237a19e8e15f2 => generated 275 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:27.620624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a936481f-65fc-4436-bc1b-1d7aa33c23c6 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:27.650110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:27.658938 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:27.707391 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f142d436-b5a6-43ea-b85e-ef3cff0f6e00 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:27.720050 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a936481f-65fc-4436-bc1b-1d7aa33c23c6 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:27.740629 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed66a57-7e64-4f8f-91ba-548d2382a8ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:27.741260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed66a57-7e64-4f8f-91ba-548d2382a8ee None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:27.741547 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed66a57-7e64-4f8f-91ba-548d2382a8ee None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:27.741819 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed66a57-7e64-4f8f-91ba-548d2382a8ee None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:27.742942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed66a57-7e64-4f8f-91ba-548d2382a8ee None admin] RBAC: Authorizing `identity:get_project(project_id=feaf606da67b4ccf9411691df09f4216)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:27.744397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed66a57-7e64-4f8f-91ba-548d2382a8ee None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:27.746042 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 46/85] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:27 2022] GET /identity/v3/projects/feaf606da67b4ccf9411691df09f4216 => generated 298 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:27.751405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed66a57-7e64-4f8f-91ba-548d2382a8ee None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:27.758688 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29aba85d-4f79-42ba-8e39-b0c60aa845f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:27.759300 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29aba85d-4f79-42ba-8e39-b0c60aa845f0 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:27.759579 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29aba85d-4f79-42ba-8e39-b0c60aa845f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:27.759854 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29aba85d-4f79-42ba-8e39-b0c60aa845f0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:27.826385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29aba85d-4f79-42ba-8e39-b0c60aa845f0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:27.827835 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29aba85d-4f79-42ba-8e39-b0c60aa845f0 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:27.845700 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed66a57-7e64-4f8f-91ba-548d2382a8ee None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:27.855851 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:27.861908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:27.879707 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fa0fe5-37b5-4115-ad43-e37aac2ad108 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:27.880306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fa0fe5-37b5-4115-ad43-e37aac2ad108 None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:27.880583 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fa0fe5-37b5-4115-ad43-e37aac2ad108 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:27.880853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fa0fe5-37b5-4115-ad43-e37aac2ad108 None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:27.881502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fa0fe5-37b5-4115-ad43-e37aac2ad108 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:27.882965 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fa0fe5-37b5-4115-ad43-e37aac2ad108 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:27.926190 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 40/86] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:42:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:42:27.927030 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29aba85d-4f79-42ba-8e39-b0c60aa845f0 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:28.007315 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 47/87] 10.222.0.62 () {60 vars in 1132 bytes} [Fri Sep 23 13:42:27 2022] POST /identity/v3/users => generated 289 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:42:28.008065 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fa0fe5-37b5-4115-ad43-e37aac2ad108 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:28.030255 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29aba85d-4f79-42ba-8e39-b0c60aa845f0 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:28.063140 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:28.069404 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042e3d58-3034-4498-96c5-e71af4a94fbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:28.070305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042e3d58-3034-4498-96c5-e71af4a94fbb None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:28.070698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042e3d58-3034-4498-96c5-e71af4a94fbb None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:28.070984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-042e3d58-3034-4498-96c5-e71af4a94fbb None admin] PATH_INFO: `/v3/users/15dbb8e9805348eb9d1237a19e8e15f2` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:28.072286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042e3d58-3034-4498-96c5-e71af4a94fbb None admin] RBAC: Authorizing `identity:get_user(user_id=15dbb8e9805348eb9d1237a19e8e15f2)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:28.073852 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-042e3d58-3034-4498-96c5-e71af4a94fbb None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:28.080868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 41/88] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:27 2022] GET /identity/v3/users/15dbb8e9805348eb9d1237a19e8e15f2 => generated 275 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:28.080868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042e3d58-3034-4498-96c5-e71af4a94fbb None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:28.136185 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fa0fe5-37b5-4115-ad43-e37aac2ad108 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:28.190159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-042e3d58-3034-4498-96c5-e71af4a94fbb None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:28.226844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a923c5d1-70ee-403a-a5f9-c38227c7c00f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:28.227472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a923c5d1-70ee-403a-a5f9-c38227c7c00f None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:28.227749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a923c5d1-70ee-403a-a5f9-c38227c7c00f None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:28.228016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a923c5d1-70ee-403a-a5f9-c38227c7c00f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:28.235435 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b852f4-3034-44c4-9937-6cf135c5847d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:28.236052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b852f4-3034-44c4-9937-6cf135c5847d None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:28.236333 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b852f4-3034-44c4-9937-6cf135c5847d None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:28.236605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b852f4-3034-44c4-9937-6cf135c5847d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:28.237329 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b852f4-3034-44c4-9937-6cf135c5847d None admin] RBAC: Adding query filter params (role.id=e8367770d3124f2db1821b6af66bfeab, scope.project.id=feaf606da67b4ccf9411691df09f4216, user.id=15dbb8e9805348eb9d1237a19e8e15f2) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:28.238087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b852f4-3034-44c4-9937-6cf135c5847d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:28.239589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b852f4-3034-44c4-9937-6cf135c5847d None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:28.247408 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 42/89] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:42:28 2022] GET /identity/v3/role_assignments?user.id=15dbb8e9805348eb9d1237a19e8e15f2&scope.project.id=feaf606da67b4ccf9411691df09f4216&role.id=e8367770d3124f2db1821b6af66bfeab => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:28.248236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b852f4-3034-44c4-9937-6cf135c5847d None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:28.279056 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:28.322751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a923c5d1-70ee-403a-a5f9-c38227c7c00f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:28.324343 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a923c5d1-70ee-403a-a5f9-c38227c7c00f None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:28.346665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:28.397614 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b852f4-3034-44c4-9937-6cf135c5847d None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:28.441936 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943e3846-d182-4d55-b95c-3efe6a487a37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:28.442744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943e3846-d182-4d55-b95c-3efe6a487a37 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:28.443148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943e3846-d182-4d55-b95c-3efe6a487a37 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:28.444703 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943e3846-d182-4d55-b95c-3efe6a487a37 None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:28.447026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943e3846-d182-4d55-b95c-3efe6a487a37 None admin] RBAC: Authorizing `identity:get_project(project_id=feaf606da67b4ccf9411691df09f4216)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:28.448490 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943e3846-d182-4d55-b95c-3efe6a487a37 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:28.453639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 43/90] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:28 2022] GET /identity/v3/projects/feaf606da67b4ccf9411691df09f4216 => generated 298 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:28.453639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943e3846-d182-4d55-b95c-3efe6a487a37 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:28.476027 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 48/91] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:42:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 468 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:42:28.477295 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a923c5d1-70ee-403a-a5f9-c38227c7c00f None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:28.568716 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:28.571283 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943e3846-d182-4d55-b95c-3efe6a487a37 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:28.602742 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:28.609532 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a923c5d1-70ee-403a-a5f9-c38227c7c00f None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:28.611182 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ef8455-47c5-4f1c-94a4-09b86ad389db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:28.613100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ef8455-47c5-4f1c-94a4-09b86ad389db None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:28.613608 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ef8455-47c5-4f1c-94a4-09b86ad389db None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:28.614086 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ef8455-47c5-4f1c-94a4-09b86ad389db None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:28.649315 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808d6746-1b9c-46ad-9e9b-4e3cd959a118 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:28.650168 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808d6746-1b9c-46ad-9e9b-4e3cd959a118 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:28.650540 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808d6746-1b9c-46ad-9e9b-4e3cd959a118 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:28.650810 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808d6746-1b9c-46ad-9e9b-4e3cd959a118 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:28.651725 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808d6746-1b9c-46ad-9e9b-4e3cd959a118 None admin] RBAC: Adding query filter params (role.id=727f0fa598e54b9a929d2cf6277c875e, scope.project.id=feaf606da67b4ccf9411691df09f4216, user.id=2ea3af238714467ab48e451a5210c19f) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:28.651979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808d6746-1b9c-46ad-9e9b-4e3cd959a118 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:28.654001 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808d6746-1b9c-46ad-9e9b-4e3cd959a118 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:28.660549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 49/92] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:42:28 2022] GET /identity/v3/role_assignments?user.id=2ea3af238714467ab48e451a5210c19f&scope.project.id=feaf606da67b4ccf9411691df09f4216&role.id=727f0fa598e54b9a929d2cf6277c875e => generated 259 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:28.661356 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808d6746-1b9c-46ad-9e9b-4e3cd959a118 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:28.707472 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ef8455-47c5-4f1c-94a4-09b86ad389db None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:28.709500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ef8455-47c5-4f1c-94a4-09b86ad389db None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:28.857164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 44/93] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:42:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:42:28.862840 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808d6746-1b9c-46ad-9e9b-4e3cd959a118 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:28.863459 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ef8455-47c5-4f1c-94a4-09b86ad389db None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:28.912957 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706808b4-ac54-4484-bf36-606a7720d55d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:28.913615 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706808b4-ac54-4484-bf36-606a7720d55d None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:28.913900 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706808b4-ac54-4484-bf36-606a7720d55d None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:28.914184 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706808b4-ac54-4484-bf36-606a7720d55d None admin] PATH_INFO: `/v3/projects/4652340ee06c4fc387b2b690cab9ee8d` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:28.915976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706808b4-ac54-4484-bf36-606a7720d55d None admin] RBAC: Authorizing `identity:get_project(project_id=4652340ee06c4fc387b2b690cab9ee8d)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:28.917441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706808b4-ac54-4484-bf36-606a7720d55d None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:28.920301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 50/94] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:28 2022] GET /identity/v3/projects/4652340ee06c4fc387b2b690cab9ee8d => generated 312 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:28.930187 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:28.930345 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706808b4-ac54-4484-bf36-606a7720d55d None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:28.938827 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:29.088946 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ef8455-47c5-4f1c-94a4-09b86ad389db None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:29.146150 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706808b4-ac54-4484-bf36-606a7720d55d None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:29.191493 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c633df5-dfd7-4697-8f16-7f147cf89989 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:29.192179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c633df5-dfd7-4697-8f16-7f147cf89989 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:29.192466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c633df5-dfd7-4697-8f16-7f147cf89989 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:29.192738 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c633df5-dfd7-4697-8f16-7f147cf89989 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:29.193479 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c633df5-dfd7-4697-8f16-7f147cf89989 None admin] RBAC: Adding query filter params (role.id=e8367770d3124f2db1821b6af66bfeab, scope.project.id=4652340ee06c4fc387b2b690cab9ee8d, user.id=15dbb8e9805348eb9d1237a19e8e15f2) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:29.193789 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c633df5-dfd7-4697-8f16-7f147cf89989 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:29.195367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c633df5-dfd7-4697-8f16-7f147cf89989 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:29.205429 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 51/95] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:42:28 2022] GET /identity/v3/role_assignments?user.id=15dbb8e9805348eb9d1237a19e8e15f2&scope.project.id=4652340ee06c4fc387b2b690cab9ee8d&role.id=e8367770d3124f2db1821b6af66bfeab => generated 259 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:29.211148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a831c3-827f-401f-83b0-be1e6bbaf770 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:29.211819 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a831c3-827f-401f-83b0-be1e6bbaf770 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:29.212096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a831c3-827f-401f-83b0-be1e6bbaf770 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:29.212362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a831c3-827f-401f-83b0-be1e6bbaf770 None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:29.219377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a831c3-827f-401f-83b0-be1e6bbaf770 None admin] RBAC: Authorizing `identity:get_project(project_id=feaf606da67b4ccf9411691df09f4216)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:29.220913 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a831c3-827f-401f-83b0-be1e6bbaf770 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:29.222957 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 45/96] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:28 2022] GET /identity/v3/projects/feaf606da67b4ccf9411691df09f4216 => generated 298 bytes in 360 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:29.228886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c633df5-dfd7-4697-8f16-7f147cf89989 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:29.340447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:29.374164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c633df5-dfd7-4697-8f16-7f147cf89989 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:29.425709 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39d2df1-6603-4a88-821f-a704a248b965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:29.426258 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39d2df1-6603-4a88-821f-a704a248b965 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:29.426567 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39d2df1-6603-4a88-821f-a704a248b965 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:29.426841 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39d2df1-6603-4a88-821f-a704a248b965 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:29.427584 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39d2df1-6603-4a88-821f-a704a248b965 None admin] RBAC: Adding query filter params (role.id=5ef6f90ed37e44adafc451b081a501d1, scope.project.id=feaf606da67b4ccf9411691df09f4216, user.id=15dbb8e9805348eb9d1237a19e8e15f2) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:29.427895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39d2df1-6603-4a88-821f-a704a248b965 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:29.429663 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39d2df1-6603-4a88-821f-a704a248b965 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:29.436976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 52/97] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:42:29 2022] GET /identity/v3/role_assignments?user.id=15dbb8e9805348eb9d1237a19e8e15f2&scope.project.id=feaf606da67b4ccf9411691df09f4216&role.id=5ef6f90ed37e44adafc451b081a501d1 => generated 259 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:30.868176 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405bc7dc-d6bb-4618-b962-bc442ee8614a None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:30.868676 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405bc7dc-d6bb-4618-b962-bc442ee8614a None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:30.868748 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405bc7dc-d6bb-4618-b962-bc442ee8614a None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:30.870420 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 46/98] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:30.876732 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47fb82c-2ebb-4592-b93c-650b1558c494 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:30.877193 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47fb82c-2ebb-4592-b93c-650b1558c494 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:30.877275 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47fb82c-2ebb-4592-b93c-650b1558c494 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:30.904458 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-f47fb82c-2ebb-4592-b93c-650b1558c494 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:30.905681 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f47fb82c-2ebb-4592-b93c-650b1558c494 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:30.992520 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 53/99] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:31.358223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e7b6e5-5d6f-4b92-bb56-763d0a9e9824 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:31.358617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e7b6e5-5d6f-4b92-bb56-763d0a9e9824 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:31.358822 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e7b6e5-5d6f-4b92-bb56-763d0a9e9824 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:31.360199 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 47/100] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:42:31.365776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628058e9-b530-4239-9728-da497c1ded52 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:31.366067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628058e9-b530-4239-9728-da497c1ded52 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:31.366359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628058e9-b530-4239-9728-da497c1ded52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:31.391417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-628058e9-b530-4239-9728-da497c1ded52 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:31.392646 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-628058e9-b530-4239-9728-da497c1ded52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:31.451279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e7b6e5-5d6f-4b92-bb56-763d0a9e9824 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:31.472143 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:31.488749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 54/101] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:31.615279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:31.620297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e7b6e5-5d6f-4b92-bb56-763d0a9e9824 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:31.668998 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a153763-787d-453f-8c59-a10b9fd9e4d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:31.670153 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a153763-787d-453f-8c59-a10b9fd9e4d6 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:31.670473 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a153763-787d-453f-8c59-a10b9fd9e4d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:31.670874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a153763-787d-453f-8c59-a10b9fd9e4d6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:31.672027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a153763-787d-453f-8c59-a10b9fd9e4d6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:31.673505 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a153763-787d-453f-8c59-a10b9fd9e4d6 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:31.679516 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 48/102] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:42:31 2022] GET /identity/v3/domains/default => generated 199 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:31.680904 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628058e9-b530-4239-9728-da497c1ded52 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:31.736997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbe159b-151f-4287-abd2-d067a50da5a1 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:31.737268 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbe159b-151f-4287-abd2-d067a50da5a1 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:31.737500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbe159b-151f-4287-abd2-d067a50da5a1 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:31.738909 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 49/103] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:42:31.743649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef05ae10-9765-4b78-aa8b-1fa70bff7be2 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:31.743936 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef05ae10-9765-4b78-aa8b-1fa70bff7be2 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:31.744216 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef05ae10-9765-4b78-aa8b-1fa70bff7be2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:31.775886 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-ef05ae10-9765-4b78-aa8b-1fa70bff7be2 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:31.777150 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-ef05ae10-9765-4b78-aa8b-1fa70bff7be2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:31.860457 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628058e9-b530-4239-9728-da497c1ded52 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:31.925097 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6b192c-0859-4250-bc99-373234da3de0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:31.926508 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6b192c-0859-4250-bc99-373234da3de0 None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:31.927003 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6b192c-0859-4250-bc99-373234da3de0 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:31.927464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6b192c-0859-4250-bc99-373234da3de0 None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:31.928399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6b192c-0859-4250-bc99-373234da3de0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:31.930232 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6b192c-0859-4250-bc99-373234da3de0 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:31.940297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 50/104] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 23 13:42:31.999209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef05ae10-9765-4b78-aa8b-1fa70bff7be2 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:32.053249 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 55/105] 10.222.0.62 () {60 vars in 1141 bytes} [Fri Sep 23 13:42:31 2022] POST /identity/v3/projects => generated 302 bytes in 373 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:42:32.122957 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:32.131254 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef05ae10-9765-4b78-aa8b-1fa70bff7be2 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:32.177251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e91dfc3-e66a-4e5f-8d17-185056508584 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:32.177941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e91dfc3-e66a-4e5f-8d17-185056508584 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:32.178197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e91dfc3-e66a-4e5f-8d17-185056508584 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:32.178499 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e91dfc3-e66a-4e5f-8d17-185056508584 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:32.204765 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e91dfc3-e66a-4e5f-8d17-185056508584 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:32.209775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-0e91dfc3-e66a-4e5f-8d17-185056508584 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 23 13:42:32.209775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:32.209775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:32.209775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:32.209775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:32.209775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:32.209775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:32.209775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:32.209775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:32.209775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:32.209775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:32.209775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:32.209775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:32.209775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:32.209775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:32.209775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:32.210657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:32.210657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:32.210657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:32.210657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:32.210657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:32.210657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:32.210657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:32.210657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:32.210657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:32.210657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:32.210657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:32.210657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:32.210657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:32.210657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:32.210657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:32.210657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:32.211416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:32.211416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:32.211416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:32.211416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:32.211416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:32.211416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:32.211416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:32.211416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:32.211416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:32.211416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:32.211416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 23 13:42:32.211416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:42:32.211416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 51/106] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:31 2022] GET /identity/v3/roles/reader => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:42:32.217846 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6b192c-0859-4250-bc99-373234da3de0 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:32.234182 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9f5064-f5eb-41ae-9c72-2c7aa15a1d5d None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:32.237189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9f5064-f5eb-41ae-9c72-2c7aa15a1d5d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:32.237480 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed9f5064-f5eb-41ae-9c72-2c7aa15a1d5d None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:32.240329 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 52/107] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:32 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:32.245351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04af1e07-456f-4c69-8d0a-0cf1ec86a84b None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:32.245651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04af1e07-456f-4c69-8d0a-0cf1ec86a84b None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:32.245912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04af1e07-456f-4c69-8d0a-0cf1ec86a84b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:32.290006 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-04af1e07-456f-4c69-8d0a-0cf1ec86a84b None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:32.291179 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-04af1e07-456f-4c69-8d0a-0cf1ec86a84b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:32.316982 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:32.409548 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6b192c-0859-4250-bc99-373234da3de0 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:32.430027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 53/108] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:32.431200 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04af1e07-456f-4c69-8d0a-0cf1ec86a84b None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:32.468992 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad34094-279c-4bbf-bb93-18ee1feed3e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:32.469619 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad34094-279c-4bbf-bb93-18ee1feed3e7 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:32.469902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad34094-279c-4bbf-bb93-18ee1feed3e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:32.470175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad34094-279c-4bbf-bb93-18ee1feed3e7 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:32.470885 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad34094-279c-4bbf-bb93-18ee1feed3e7 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:32.471187 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad34094-279c-4bbf-bb93-18ee1feed3e7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:32.472643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad34094-279c-4bbf-bb93-18ee1feed3e7 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:32.484752 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 56/109] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:42:32 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:32.491799 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad34094-279c-4bbf-bb93-18ee1feed3e7 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:32.602301 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:32.631808 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04af1e07-456f-4c69-8d0a-0cf1ec86a84b None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:32.673298 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad34094-279c-4bbf-bb93-18ee1feed3e7 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:32.688977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:32.694095 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4482aa19-ddcf-4fa7-9538-4600c3ace70b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:32.696945 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4482aa19-ddcf-4fa7-9538-4600c3ace70b None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:32.696945 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4482aa19-ddcf-4fa7-9538-4600c3ace70b None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:32.696945 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4482aa19-ddcf-4fa7-9538-4600c3ace70b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:32.708578 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de90244-4400-4c5e-9ebf-9ce9c6586327 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:32.709220 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de90244-4400-4c5e-9ebf-9ce9c6586327 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:32.709603 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de90244-4400-4c5e-9ebf-9ce9c6586327 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:32.709885 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de90244-4400-4c5e-9ebf-9ce9c6586327 None admin] PATH_INFO: `/v3/users/7a356c122f764e09b20f6bb7d3a6e028` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:32.716892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4482aa19-ddcf-4fa7-9538-4600c3ace70b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:32.721832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-4482aa19-ddcf-4fa7-9538-4600c3ace70b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:32.721832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:32.721832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:32.721832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:32.721832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:32.721832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:32.721832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:32.721832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:32.721832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:32.721832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:32.721832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:32.721832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:32.721832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:32.721832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:32.721832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:32.721832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:32.722671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:32.722671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:32.722671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:32.722671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:32.722671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:32.722671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:32.722671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:32.722671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:32.722671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:32.722671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:32.722671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:32.722671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:32.722671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:32.722671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:32.722671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:32.722671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:32.723433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:32.723433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:32.723433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:32.723433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:32.723433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:32.723433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:32.723433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:32.723433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:32.723433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:32.723433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:32.723433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:32.723433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:42:32.727502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de90244-4400-4c5e-9ebf-9ce9c6586327 None admin] RBAC: Authorizing `identity:get_user(user_id=7a356c122f764e09b20f6bb7d3a6e028)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:32.728996 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de90244-4400-4c5e-9ebf-9ce9c6586327 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:32.731370 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 57/110] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:32 2022] GET /identity/v3/users/7a356c122f764e09b20f6bb7d3a6e028 => generated 289 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:32.731979 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 54/111] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:32 2022] GET /identity/v3/roles/member => generated 84 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:32.735607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d11eb39-0828-4f8e-a62c-3eec20b25de5 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:32.735901 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d11eb39-0828-4f8e-a62c-3eec20b25de5 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:32.736183 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d11eb39-0828-4f8e-a62c-3eec20b25de5 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:32.739134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4482aa19-ddcf-4fa7-9538-4600c3ace70b None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:32.739713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 58/112] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:32 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:32.740532 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d11eb39-0828-4f8e-a62c-3eec20b25de5 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:32.865304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d11eb39-0828-4f8e-a62c-3eec20b25de5 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:32.879714 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4482aa19-ddcf-4fa7-9538-4600c3ace70b None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:32.909665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d95c7f-8992-4f22-a01b-5c113d329438 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:32.910556 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d95c7f-8992-4f22-a01b-5c113d329438 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:32.910839 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d95c7f-8992-4f22-a01b-5c113d329438 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:32.911579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d95c7f-8992-4f22-a01b-5c113d329438 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:32.923721 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2275b37-36af-4a5b-9f77-6d061bb6fc20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:32.924333 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2275b37-36af-4a5b-9f77-6d061bb6fc20 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:32.924614 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2275b37-36af-4a5b-9f77-6d061bb6fc20 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:32.924879 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2275b37-36af-4a5b-9f77-6d061bb6fc20 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:32.925934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2275b37-36af-4a5b-9f77-6d061bb6fc20 None admin] RBAC: Adding query filter params (name=member) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:32.926236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2275b37-36af-4a5b-9f77-6d061bb6fc20 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:32.927995 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2275b37-36af-4a5b-9f77-6d061bb6fc20 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:32.937156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 55/113] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:42:32 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:32.940228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8536b624-9c88-4430-979b-c1aa3a22253c None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:32.940722 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8536b624-9c88-4430-979b-c1aa3a22253c None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:32.941787 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8536b624-9c88-4430-979b-c1aa3a22253c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:32.948280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:32.964240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-8536b624-9c88-4430-979b-c1aa3a22253c None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:32.966215 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8536b624-9c88-4430-979b-c1aa3a22253c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:32.968461 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:32.989460 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d95c7f-8992-4f22-a01b-5c113d329438 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:32.990958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d95c7f-8992-4f22-a01b-5c113d329438 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:33.046397 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 56/114] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:33.046397 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8536b624-9c88-4430-979b-c1aa3a22253c None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:33.122134 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 59/115] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:42:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:42:33.123634 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0cb44a-ad0f-46df-931a-80b29cc291ee None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:33.124063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0cb44a-ad0f-46df-931a-80b29cc291ee None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:33.124476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0cb44a-ad0f-46df-931a-80b29cc291ee None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:33.126357 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 60/116] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:33.127206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0cb44a-ad0f-46df-931a-80b29cc291ee None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:33.163361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8536b624-9c88-4430-979b-c1aa3a22253c None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:33.198819 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:33.201354 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6974a10d-de77-428c-908b-f01f375d6545 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:33.202214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6974a10d-de77-428c-908b-f01f375d6545 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:33.202680 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6974a10d-de77-428c-908b-f01f375d6545 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:33.203149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6974a10d-de77-428c-908b-f01f375d6545 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:33.210808 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:33.215790 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6974a10d-de77-428c-908b-f01f375d6545 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:33.221684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-6974a10d-de77-428c-908b-f01f375d6545 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:42:33.221684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:33.221684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:33.221684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:33.221684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:33.221684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:33.221684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:33.221684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:33.221684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:33.221684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:33.221684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:33.221684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:33.221684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:33.221684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:33.221684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:33.221684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:33.223294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:33.223294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:33.223294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:33.223294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:33.223294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:33.223294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:33.223294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:33.223294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:33.223294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:33.223294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:33.223294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:33.223294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:33.223294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:33.223294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:33.223294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:33.223294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:33.224233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:33.224233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:33.224233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:33.224233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:33.224233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:33.224233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:33.224233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:33.224233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:33.224233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:33.224233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:33.224233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:42:33.224233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:42:33.225752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 57/117] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:42:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:33.232006 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6974a10d-de77-428c-908b-f01f375d6545 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:33.242778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0cb44a-ad0f-46df-931a-80b29cc291ee None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:33.277389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c828b61-a4fa-4062-a5a3-da84b8e0fbc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:33.278676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c828b61-a4fa-4062-a5a3-da84b8e0fbc0 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:33.279071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c828b61-a4fa-4062-a5a3-da84b8e0fbc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:33.279412 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c828b61-a4fa-4062-a5a3-da84b8e0fbc0 None admin] PATH_INFO: `/v3/users/15dbb8e9805348eb9d1237a19e8e15f2` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:33.280899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c828b61-a4fa-4062-a5a3-da84b8e0fbc0 None admin] RBAC: Authorizing `identity:get_user(user_id=15dbb8e9805348eb9d1237a19e8e15f2)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:33.282454 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c828b61-a4fa-4062-a5a3-da84b8e0fbc0 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:33.284137 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 61/118] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:33 2022] GET /identity/v3/users/15dbb8e9805348eb9d1237a19e8e15f2 => generated 275 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:33.288705 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d23b7ed-ef7e-43fa-b4cf-1d5ffac28807 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:33.288705 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d23b7ed-ef7e-43fa-b4cf-1d5ffac28807 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:33.288705 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d23b7ed-ef7e-43fa-b4cf-1d5ffac28807 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:33.304826 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-4d23b7ed-ef7e-43fa-b4cf-1d5ffac28807 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:33.306102 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-4d23b7ed-ef7e-43fa-b4cf-1d5ffac28807 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:33.352414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6974a10d-de77-428c-908b-f01f375d6545 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:33.377254 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 62/119] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:33.378521 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d23b7ed-ef7e-43fa-b4cf-1d5ffac28807 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:33.393684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a2597c-7ff7-4d4b-8ffa-85cfd5961a6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:33.394769 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a2597c-7ff7-4d4b-8ffa-85cfd5961a6d None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:33.395127 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a2597c-7ff7-4d4b-8ffa-85cfd5961a6d None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:33.395515 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a2597c-7ff7-4d4b-8ffa-85cfd5961a6d None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:33.396886 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a2597c-7ff7-4d4b-8ffa-85cfd5961a6d None admin] RBAC: Authorizing `identity:get_project(project_id=feaf606da67b4ccf9411691df09f4216)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:33.398976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a2597c-7ff7-4d4b-8ffa-85cfd5961a6d None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:33.402016 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 58/120] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:33 2022] GET /identity/v3/projects/feaf606da67b4ccf9411691df09f4216 => generated 298 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:33.403505 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a2597c-7ff7-4d4b-8ffa-85cfd5961a6d None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:33.446856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:33.487096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:33.499160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d23b7ed-ef7e-43fa-b4cf-1d5ffac28807 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:33.521529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a2597c-7ff7-4d4b-8ffa-85cfd5961a6d None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:33.546364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bdd420-0043-497a-8065-3e7f6bf3da68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:33.547326 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bdd420-0043-497a-8065-3e7f6bf3da68 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:33.547618 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bdd420-0043-497a-8065-3e7f6bf3da68 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:33.548314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bdd420-0043-497a-8065-3e7f6bf3da68 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:33.549146 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bdd420-0043-497a-8065-3e7f6bf3da68 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:33.549577 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bdd420-0043-497a-8065-3e7f6bf3da68 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:33.551155 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6bdd420-0043-497a-8065-3e7f6bf3da68 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:33.561983 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d489ed13-aed7-4b45-9d6b-ad50327b3d3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:33.562644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d489ed13-aed7-4b45-9d6b-ad50327b3d3c None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:33.563060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d489ed13-aed7-4b45-9d6b-ad50327b3d3c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:33.563250 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d489ed13-aed7-4b45-9d6b-ad50327b3d3c None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:33.566612 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 63/121] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:33.568172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bdd420-0043-497a-8065-3e7f6bf3da68 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:33.568524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d489ed13-aed7-4b45-9d6b-ad50327b3d3c None admin] RBAC: Authorizing `identity:get_project(project_id=feaf606da67b4ccf9411691df09f4216)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:33.570202 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d489ed13-aed7-4b45-9d6b-ad50327b3d3c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:33.573179 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 59/122] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:33 2022] GET /identity/v3/projects/feaf606da67b4ccf9411691df09f4216 => generated 298 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:33.573840 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d489ed13-aed7-4b45-9d6b-ad50327b3d3c None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:33.676592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bdd420-0043-497a-8065-3e7f6bf3da68 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:33.696657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d489ed13-aed7-4b45-9d6b-ad50327b3d3c None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:33.701628 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:33.712111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4092ac70-36b3-4959-968a-b83191860965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:33.712744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4092ac70-36b3-4959-968a-b83191860965 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:33.713026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4092ac70-36b3-4959-968a-b83191860965 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:33.713297 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4092ac70-36b3-4959-968a-b83191860965 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:33.724690 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4092ac70-36b3-4959-968a-b83191860965 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:33.729605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-4092ac70-36b3-4959-968a-b83191860965 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:42:33.729605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:33.729605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:33.729605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:33.729605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:33.729605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:33.729605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:33.729605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:33.729605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:33.729605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:33.729605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:33.729605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:33.729605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:33.729605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:33.729605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:33.729605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:33.730598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:33.730598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:33.730598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:33.730598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:33.730598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:33.730598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:33.730598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:33.730598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:33.730598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:33.730598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:33.730598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:33.730598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:33.730598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:33.730598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:33.730598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:33.730598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:33.735633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:33.735633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:33.735633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:33.735633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:33.735633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:33.735633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:33.735633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:33.735633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:33.735633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:33.735633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:33.735633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:42:33.735633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:42:33.735633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 64/123] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:42:33 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:33.735633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4092ac70-36b3-4959-968a-b83191860965 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:33.741610 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3815094e-d7ab-4aa3-8497-5c72abce61a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:33.742243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3815094e-d7ab-4aa3-8497-5c72abce61a3 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:33.742563 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3815094e-d7ab-4aa3-8497-5c72abce61a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:33.742838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3815094e-d7ab-4aa3-8497-5c72abce61a3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:33.743579 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3815094e-d7ab-4aa3-8497-5c72abce61a3 None admin] RBAC: Adding query filter params (role.id=2c1b779eac624a728f7e590135c57450, scope.project.id=feaf606da67b4ccf9411691df09f4216, user.id=7a356c122f764e09b20f6bb7d3a6e028) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:33.743889 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3815094e-d7ab-4aa3-8497-5c72abce61a3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:33.745349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3815094e-d7ab-4aa3-8497-5c72abce61a3 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:33.748044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:33.752715 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 60/124] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:42:33 2022] GET /identity/v3/role_assignments?user.id=7a356c122f764e09b20f6bb7d3a6e028&scope.project.id=feaf606da67b4ccf9411691df09f4216&role.id=2c1b779eac624a728f7e590135c57450 => generated 259 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:33.753398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3815094e-d7ab-4aa3-8497-5c72abce61a3 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:33.861199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4092ac70-36b3-4959-968a-b83191860965 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:33.876816 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3815094e-d7ab-4aa3-8497-5c72abce61a3 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:33.905139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110c0e25-1896-43b9-b24f-25eff9d915b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:33.906142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110c0e25-1896-43b9-b24f-25eff9d915b8 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:33.906649 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110c0e25-1896-43b9-b24f-25eff9d915b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:33.907244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110c0e25-1896-43b9-b24f-25eff9d915b8 None admin] PATH_INFO: `/v3/users/2ea3af238714467ab48e451a5210c19f` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:33.909321 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110c0e25-1896-43b9-b24f-25eff9d915b8 None admin] RBAC: Authorizing `identity:get_user(user_id=2ea3af238714467ab48e451a5210c19f)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:33.911038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110c0e25-1896-43b9-b24f-25eff9d915b8 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:33.912995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 65/125] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:33 2022] GET /identity/v3/users/2ea3af238714467ab48e451a5210c19f => generated 247 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:33.913960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110c0e25-1896-43b9-b24f-25eff9d915b8 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:33.922293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1755b5d3-564b-4aab-8f21-6fee3b0c26b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:33.922529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1755b5d3-564b-4aab-8f21-6fee3b0c26b8 None admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:33.922812 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1755b5d3-564b-4aab-8f21-6fee3b0c26b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:33.923155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1755b5d3-564b-4aab-8f21-6fee3b0c26b8 None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216/users/15dbb8e9805348eb9d1237a19e8e15f2/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:33.925340 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1755b5d3-564b-4aab-8f21-6fee3b0c26b8 None admin] RBAC: Authorizing `identity:create_grant(project_id=feaf606da67b4ccf9411691df09f4216, user_id=15dbb8e9805348eb9d1237a19e8e15f2, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:33.926917 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1755b5d3-564b-4aab-8f21-6fee3b0c26b8 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:33.980263 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:33.997958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 61/126] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:42:33 2022] PUT /identity/v3/projects/feaf606da67b4ccf9411691df09f4216/users/15dbb8e9805348eb9d1237a19e8e15f2/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:42:33.999395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1755b5d3-564b-4aab-8f21-6fee3b0c26b8 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:34.056420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110c0e25-1896-43b9-b24f-25eff9d915b8 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:34.068984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:34.119188 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1755b5d3-564b-4aab-8f21-6fee3b0c26b8 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:34.123069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328ec10d-f54a-4639-a23b-182ae1a461c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:34.123891 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328ec10d-f54a-4639-a23b-182ae1a461c4 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:34.124275 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328ec10d-f54a-4639-a23b-182ae1a461c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:34.124683 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328ec10d-f54a-4639-a23b-182ae1a461c4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:34.144051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328ec10d-f54a-4639-a23b-182ae1a461c4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:34.154979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-328ec10d-f54a-4639-a23b-182ae1a461c4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:34.154979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:34.154979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:34.154979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:34.154979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:34.154979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:34.154979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:34.154979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:34.154979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:34.154979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:34.154979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:34.154979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:34.154979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:34.154979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:34.154979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:34.154979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:34.155916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:34.155916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:34.155916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:34.155916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:34.155916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:34.155916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:34.155916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:34.155916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:34.155916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:34.155916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:34.155916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:34.155916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:34.155916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:34.155916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:34.155916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:34.155916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:34.156677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:34.156677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:34.156677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:34.156677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:34.156677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:34.156677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:34.156677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:34.156677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:34.156677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:34.156677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:34.156677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:34.156677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:42:34.156677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 66/127] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:34.157338 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328ec10d-f54a-4639-a23b-182ae1a461c4 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:34.165403 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1805c0bc-2d1f-408c-be93-d3c5a802dd20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:34.166294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1805c0bc-2d1f-408c-be93-d3c5a802dd20 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:34.166818 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1805c0bc-2d1f-408c-be93-d3c5a802dd20 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:34.167277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1805c0bc-2d1f-408c-be93-d3c5a802dd20 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:34.168367 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1805c0bc-2d1f-408c-be93-d3c5a802dd20 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:34.168873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1805c0bc-2d1f-408c-be93-d3c5a802dd20 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:34.170814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1805c0bc-2d1f-408c-be93-d3c5a802dd20 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:34.182301 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 62/128] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:42:33 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:34.183486 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1805c0bc-2d1f-408c-be93-d3c5a802dd20 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:34.347594 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1805c0bc-2d1f-408c-be93-d3c5a802dd20 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:34.363840 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328ec10d-f54a-4639-a23b-182ae1a461c4 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:34.373052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:34.380263 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:34.391577 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1388db40-176f-414f-8c0c-5e67957dc177 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:34.392493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1388db40-176f-414f-8c0c-5e67957dc177 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:34.392947 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1388db40-176f-414f-8c0c-5e67957dc177 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:34.393387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1388db40-176f-414f-8c0c-5e67957dc177 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:34.394251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1388db40-176f-414f-8c0c-5e67957dc177 None admin] RBAC: Adding query filter params (name=member) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:34.394778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1388db40-176f-414f-8c0c-5e67957dc177 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:34.396388 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1388db40-176f-414f-8c0c-5e67957dc177 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:34.403487 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857d903b-77a9-4d6b-b80e-bd2a404f1b91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:34.404136 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d903b-77a9-4d6b-b80e-bd2a404f1b91 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:34.404421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d903b-77a9-4d6b-b80e-bd2a404f1b91 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:34.404701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857d903b-77a9-4d6b-b80e-bd2a404f1b91 None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:34.405977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857d903b-77a9-4d6b-b80e-bd2a404f1b91 None admin] RBAC: Authorizing `identity:get_project(project_id=feaf606da67b4ccf9411691df09f4216)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:34.407488 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857d903b-77a9-4d6b-b80e-bd2a404f1b91 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:34.408175 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 63/129] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:42:34 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:34.409192 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1388db40-176f-414f-8c0c-5e67957dc177 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:34.409303 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 67/130] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:34 2022] GET /identity/v3/projects/feaf606da67b4ccf9411691df09f4216 => generated 298 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:34.414628 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857d903b-77a9-4d6b-b80e-bd2a404f1b91 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:34.531083 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857d903b-77a9-4d6b-b80e-bd2a404f1b91 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:34.567309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a8fbbd-7382-481a-a5bf-e4d81f44a66e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:34.567959 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a8fbbd-7382-481a-a5bf-e4d81f44a66e None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:34.568249 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a8fbbd-7382-481a-a5bf-e4d81f44a66e None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:34.568530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0a8fbbd-7382-481a-a5bf-e4d81f44a66e None admin] PATH_INFO: `/v3/users/15dbb8e9805348eb9d1237a19e8e15f2` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:34.569693 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a8fbbd-7382-481a-a5bf-e4d81f44a66e None admin] RBAC: Authorizing `identity:get_user(user_id=15dbb8e9805348eb9d1237a19e8e15f2)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:34.571314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0a8fbbd-7382-481a-a5bf-e4d81f44a66e None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:34.572915 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 68/131] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:34 2022] GET /identity/v3/users/15dbb8e9805348eb9d1237a19e8e15f2 => generated 275 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:34.574085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a8fbbd-7382-481a-a5bf-e4d81f44a66e None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:34.633587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1388db40-176f-414f-8c0c-5e67957dc177 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:34.642736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:34.686406 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0a8fbbd-7382-481a-a5bf-e4d81f44a66e None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:34.723155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6486e61f-3bc3-44e5-8c1f-7f4c5553c2d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:34.723817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6486e61f-3bc3-44e5-8c1f-7f4c5553c2d5 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:34.724136 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6486e61f-3bc3-44e5-8c1f-7f4c5553c2d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:34.724418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6486e61f-3bc3-44e5-8c1f-7f4c5553c2d5 None admin] PATH_INFO: `/v3/users/15dbb8e9805348eb9d1237a19e8e15f2` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:34.731308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6486e61f-3bc3-44e5-8c1f-7f4c5553c2d5 None admin] RBAC: Authorizing `identity:get_user(user_id=15dbb8e9805348eb9d1237a19e8e15f2)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:34.732932 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6486e61f-3bc3-44e5-8c1f-7f4c5553c2d5 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:34.735299 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 64/132] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:34 2022] GET /identity/v3/users/15dbb8e9805348eb9d1237a19e8e15f2 => generated 275 bytes in 326 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:34.735859 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd15b424-2646-4f9e-a7d4-66439234d522 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:34.736111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6486e61f-3bc3-44e5-8c1f-7f4c5553c2d5 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:34.736604 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd15b424-2646-4f9e-a7d4-66439234d522 None admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:34.736917 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd15b424-2646-4f9e-a7d4-66439234d522 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:34.737162 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd15b424-2646-4f9e-a7d4-66439234d522 None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216/users/2ea3af238714467ab48e451a5210c19f/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:34.739183 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd15b424-2646-4f9e-a7d4-66439234d522 None admin] RBAC: Authorizing `identity:create_grant(project_id=feaf606da67b4ccf9411691df09f4216, user_id=2ea3af238714467ab48e451a5210c19f, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:34.740615 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd15b424-2646-4f9e-a7d4-66439234d522 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:34.801684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:34.842036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 69/133] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:42:34 2022] PUT /identity/v3/projects/feaf606da67b4ccf9411691df09f4216/users/2ea3af238714467ab48e451a5210c19f/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:42:34.842997 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd15b424-2646-4f9e-a7d4-66439234d522 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:34.857080 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6486e61f-3bc3-44e5-8c1f-7f4c5553c2d5 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:34.928519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a1ee6c-e833-40a3-bef1-fa1d839aa27c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:34.929182 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a1ee6c-e833-40a3-bef1-fa1d839aa27c None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:34.929465 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a1ee6c-e833-40a3-bef1-fa1d839aa27c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:34.929739 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a1ee6c-e833-40a3-bef1-fa1d839aa27c None admin] PATH_INFO: `/v3/projects/4652340ee06c4fc387b2b690cab9ee8d` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:34.933455 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a1ee6c-e833-40a3-bef1-fa1d839aa27c None admin] RBAC: Authorizing `identity:get_project(project_id=4652340ee06c4fc387b2b690cab9ee8d)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:34.935036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0a1ee6c-e833-40a3-bef1-fa1d839aa27c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:34.940630 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 65/134] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:34 2022] GET /identity/v3/projects/4652340ee06c4fc387b2b690cab9ee8d => generated 312 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:34.943301 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d713419-4585-4b24-a986-55a27a33ef2d None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:34.943589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d713419-4585-4b24-a986-55a27a33ef2d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:34.943868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d713419-4585-4b24-a986-55a27a33ef2d None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:34.947340 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 66/135] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:34.948191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d713419-4585-4b24-a986-55a27a33ef2d None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:35.038195 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd15b424-2646-4f9e-a7d4-66439234d522 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:35.054808 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:35.096852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598ac0e0-6ba4-4634-8e8b-192f91661f73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:35.097448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598ac0e0-6ba4-4634-8e8b-192f91661f73 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:35.097728 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598ac0e0-6ba4-4634-8e8b-192f91661f73 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:35.098001 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598ac0e0-6ba4-4634-8e8b-192f91661f73 None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:35.099206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598ac0e0-6ba4-4634-8e8b-192f91661f73 None admin] RBAC: Authorizing `identity:get_project(project_id=feaf606da67b4ccf9411691df09f4216)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:35.099768 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d713419-4585-4b24-a986-55a27a33ef2d None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:35.100731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598ac0e0-6ba4-4634-8e8b-192f91661f73 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:35.102580 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 70/136] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:34 2022] GET /identity/v3/projects/feaf606da67b4ccf9411691df09f4216 => generated 298 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:35.103884 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25363a6e-3213-47da-b494-b3a86b61f1c0 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:35.107259 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25363a6e-3213-47da-b494-b3a86b61f1c0 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:35.107259 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25363a6e-3213-47da-b494-b3a86b61f1c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:35.125270 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:35.128785 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-25363a6e-3213-47da-b494-b3a86b61f1c0 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:35.130093 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-25363a6e-3213-47da-b494-b3a86b61f1c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:35.144991 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30002764-00e5-4918-b79f-3abacfee0668 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:35.145664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30002764-00e5-4918-b79f-3abacfee0668 None admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:35.145942 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30002764-00e5-4918-b79f-3abacfee0668 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:35.146218 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30002764-00e5-4918-b79f-3abacfee0668 None admin] PATH_INFO: `/v3/projects/4652340ee06c4fc387b2b690cab9ee8d/users/15dbb8e9805348eb9d1237a19e8e15f2/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:35.149113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30002764-00e5-4918-b79f-3abacfee0668 None admin] RBAC: Authorizing `identity:create_grant(project_id=4652340ee06c4fc387b2b690cab9ee8d, user_id=15dbb8e9805348eb9d1237a19e8e15f2, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:35.150615 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30002764-00e5-4918-b79f-3abacfee0668 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:35.235420 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 67/137] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:42:34 2022] PUT /identity/v3/projects/4652340ee06c4fc387b2b690cab9ee8d/users/15dbb8e9805348eb9d1237a19e8e15f2/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:42:35.235889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 71/138] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:35.236554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30002764-00e5-4918-b79f-3abacfee0668 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:35.362108 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30002764-00e5-4918-b79f-3abacfee0668 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:35.401773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54a0b59-29c2-43e8-a2a6-2bdb5bc4ccfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:35.402997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54a0b59-29c2-43e8-a2a6-2bdb5bc4ccfe None admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:35.403281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54a0b59-29c2-43e8-a2a6-2bdb5bc4ccfe None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:35.403554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54a0b59-29c2-43e8-a2a6-2bdb5bc4ccfe None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216/users/15dbb8e9805348eb9d1237a19e8e15f2/roles/5ef6f90ed37e44adafc451b081a501d1` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:35.405758 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54a0b59-29c2-43e8-a2a6-2bdb5bc4ccfe None admin] RBAC: Authorizing `identity:create_grant(project_id=feaf606da67b4ccf9411691df09f4216, user_id=15dbb8e9805348eb9d1237a19e8e15f2, role_id=5ef6f90ed37e44adafc451b081a501d1)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:35.407306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54a0b59-29c2-43e8-a2a6-2bdb5bc4ccfe None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:35.482370 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 68/139] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:42:35 2022] PUT /identity/v3/projects/feaf606da67b4ccf9411691df09f4216/users/15dbb8e9805348eb9d1237a19e8e15f2/roles/5ef6f90ed37e44adafc451b081a501d1 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:42:35.604693 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25363a6e-3213-47da-b494-b3a86b61f1c0 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:35.717757 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25363a6e-3213-47da-b494-b3a86b61f1c0 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:35.728084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:35.758380 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7a1ae3-cec6-473c-bb25-10e937d53540 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:35.759013 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7a1ae3-cec6-473c-bb25-10e937d53540 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:35.759294 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7a1ae3-cec6-473c-bb25-10e937d53540 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:35.759579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7a1ae3-cec6-473c-bb25-10e937d53540 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:35.760733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7a1ae3-cec6-473c-bb25-10e937d53540 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:35.764848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7a1ae3-cec6-473c-bb25-10e937d53540 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:35.766586 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 72/140] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:42:35 2022] GET /identity/v3/domains/default => generated 199 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:35.772158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54a0b59-29c2-43e8-a2a6-2bdb5bc4ccfe None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:35.838023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:35.920086 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54a0b59-29c2-43e8-a2a6-2bdb5bc4ccfe None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:35.960331 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6ac404-78d7-479a-98df-015b6adc5e78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:35.960965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6ac404-78d7-479a-98df-015b6adc5e78 None admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:35.961248 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6ac404-78d7-479a-98df-015b6adc5e78 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:35.961528 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6ac404-78d7-479a-98df-015b6adc5e78 None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:35.962190 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6ac404-78d7-479a-98df-015b6adc5e78 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:35.963664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6ac404-78d7-479a-98df-015b6adc5e78 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:36.075993 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 69/141] 10.222.0.62 () {60 vars in 1132 bytes} [Fri Sep 23 13:42:35 2022] POST /identity/v3/users => generated 283 bytes in 304 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:42:36.521115 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaadfa5e-bf0a-4978-b545-69483797d8ea None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:36.521977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaadfa5e-bf0a-4978-b545-69483797d8ea None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:36.522411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaadfa5e-bf0a-4978-b545-69483797d8ea None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:36.523885 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 73/142] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:36.529306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b7d43c-2ed5-4443-a952-c2668cb9bfb8 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:36.529702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b7d43c-2ed5-4443-a952-c2668cb9bfb8 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:36.529881 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b7d43c-2ed5-4443-a952-c2668cb9bfb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:36.551864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-05b7d43c-2ed5-4443-a952-c2668cb9bfb8 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:36.553068 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-05b7d43c-2ed5-4443-a952-c2668cb9bfb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:36.635033 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 70/143] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:36.965378 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e6b55a-0c22-4aca-830f-6d40e0d0ef0e None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:36.966009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e6b55a-0c22-4aca-830f-6d40e0d0ef0e None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:36.966534 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e6b55a-0c22-4aca-830f-6d40e0d0ef0e None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:36.968094 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 74/144] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:36.973908 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f584ec-1084-4c82-9a1e-5c6c08ee5d8d None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:36.974196 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f584ec-1084-4c82-9a1e-5c6c08ee5d8d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:36.974510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f584ec-1084-4c82-9a1e-5c6c08ee5d8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:36.997311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-b0f584ec-1084-4c82-9a1e-5c6c08ee5d8d None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:36.998647 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-b0f584ec-1084-4c82-9a1e-5c6c08ee5d8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:37.024341 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:37.079907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 71/145] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:37.099963 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e6b55a-0c22-4aca-830f-6d40e0d0ef0e None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:37.228321 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e6b55a-0c22-4aca-830f-6d40e0d0ef0e None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:37.272905 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b31250-d870-446d-8795-ca9e6748767e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:37.273534 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b31250-d870-446d-8795-ca9e6748767e None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:37.273814 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b31250-d870-446d-8795-ca9e6748767e None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:37.274081 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b31250-d870-446d-8795-ca9e6748767e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:37.288942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b31250-d870-446d-8795-ca9e6748767e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:37.294579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-d1b31250-d870-446d-8795-ca9e6748767e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 23 13:42:37.294579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:37.294579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:37.294579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:37.294579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:37.294579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:37.294579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:37.294579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:37.294579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:37.294579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:37.294579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:37.294579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:37.294579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:37.294579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:37.294579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:37.294579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:37.295404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:37.295404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:37.295404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:37.295404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:37.295404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:37.295404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:37.295404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:37.295404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:37.295404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:37.295404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:37.295404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:37.295404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:37.295404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:37.295404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:37.295404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:37.295404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:37.296161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:37.296161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:37.296161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:37.296161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:37.296161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:37.296161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:37.296161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:37.296161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:37.296161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:37.296161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:37.296161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 23 13:42:37.296161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:42:37.296161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 75/146] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:37 2022] GET /identity/v3/roles/reader => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:37.301733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f584ec-1084-4c82-9a1e-5c6c08ee5d8d None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:37.408429 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b31250-d870-446d-8795-ca9e6748767e None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:37.421662 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f584ec-1084-4c82-9a1e-5c6c08ee5d8d None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:37.481084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:37.482535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37103b92-7a85-4669-914a-00a0a4083e5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:37.483156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37103b92-7a85-4669-914a-00a0a4083e5e None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:37.483439 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37103b92-7a85-4669-914a-00a0a4083e5e None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:37.483709 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37103b92-7a85-4669-914a-00a0a4083e5e None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:37.484400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37103b92-7a85-4669-914a-00a0a4083e5e None admin] RBAC: Adding query filter params (name=reader) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:37.484696 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37103b92-7a85-4669-914a-00a0a4083e5e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:37.486082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37103b92-7a85-4669-914a-00a0a4083e5e None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:37.502303 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 72/147] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:42:37 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:37.512210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37103b92-7a85-4669-914a-00a0a4083e5e None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:37.545167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:37.592211 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b31250-d870-446d-8795-ca9e6748767e None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:37.641037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495df026-6eb5-431a-b0ee-e33f5bb1cfd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:37.641688 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495df026-6eb5-431a-b0ee-e33f5bb1cfd1 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:37.641967 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495df026-6eb5-431a-b0ee-e33f5bb1cfd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:37.642236 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495df026-6eb5-431a-b0ee-e33f5bb1cfd1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:37.660557 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495df026-6eb5-431a-b0ee-e33f5bb1cfd1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:37.665441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-495df026-6eb5-431a-b0ee-e33f5bb1cfd1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:37.665441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:37.665441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:37.665441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:37.665441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:37.665441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:37.665441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:37.665441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:37.665441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:37.665441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:37.665441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:37.665441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:37.665441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:37.665441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:37.665441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:37.665441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:37.666251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:37.666251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:37.666251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:37.666251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:37.666251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:37.666251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:37.666251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:37.666251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:37.666251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:37.666251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:37.666251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:37.666251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:37.666251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:37.666251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:37.666251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:37.666251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:37.667122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:37.667122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:37.667122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:37.667122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:37.667122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:37.667122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:37.667122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:37.667122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:37.667122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:37.667122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:37.667122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:37.667122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:42:37.673627 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 76/148] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:42:37.679696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495df026-6eb5-431a-b0ee-e33f5bb1cfd1 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:37.731000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37103b92-7a85-4669-914a-00a0a4083e5e None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:37.795118 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bc4ee5-ce95-4556-82a3-ba129479e21d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:37.795692 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bc4ee5-ce95-4556-82a3-ba129479e21d None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:37.795972 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bc4ee5-ce95-4556-82a3-ba129479e21d None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:37.796243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bc4ee5-ce95-4556-82a3-ba129479e21d None admin] PATH_INFO: `/v3/users/7a356c122f764e09b20f6bb7d3a6e028` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:37.798430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bc4ee5-ce95-4556-82a3-ba129479e21d None admin] RBAC: Authorizing `identity:get_user(user_id=7a356c122f764e09b20f6bb7d3a6e028)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:37.799904 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bc4ee5-ce95-4556-82a3-ba129479e21d None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:37.801657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 73/149] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:37 2022] GET /identity/v3/users/7a356c122f764e09b20f6bb7d3a6e028 => generated 289 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:37.832093 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bc4ee5-ce95-4556-82a3-ba129479e21d None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:37.897249 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:37.917370 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495df026-6eb5-431a-b0ee-e33f5bb1cfd1 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:37.989524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:38.007459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381d287f-8a8b-4cf7-a4ae-53d24e5b36ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:38.008086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381d287f-8a8b-4cf7-a4ae-53d24e5b36ab None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:38.008366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381d287f-8a8b-4cf7-a4ae-53d24e5b36ab None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:38.008643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381d287f-8a8b-4cf7-a4ae-53d24e5b36ab None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:38.009130 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bc4ee5-ce95-4556-82a3-ba129479e21d None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:38.009307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381d287f-8a8b-4cf7-a4ae-53d24e5b36ab None admin] RBAC: Adding query filter params (name=member) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:38.009613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381d287f-8a8b-4cf7-a4ae-53d24e5b36ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:38.011032 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381d287f-8a8b-4cf7-a4ae-53d24e5b36ab None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:38.036658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 77/150] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:42:37 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 362 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:38.047705 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381d287f-8a8b-4cf7-a4ae-53d24e5b36ab None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:38.073328 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6091e871-086b-4349-80e9-bb6650a11cd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:38.073973 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6091e871-086b-4349-80e9-bb6650a11cd9 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:38.074829 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6091e871-086b-4349-80e9-bb6650a11cd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:38.075102 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6091e871-086b-4349-80e9-bb6650a11cd9 None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:38.078078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6091e871-086b-4349-80e9-bb6650a11cd9 None admin] RBAC: Authorizing `identity:get_project(project_id=feaf606da67b4ccf9411691df09f4216)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:38.079865 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6091e871-086b-4349-80e9-bb6650a11cd9 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:38.082105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 74/151] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:37 2022] GET /identity/v3/projects/feaf606da67b4ccf9411691df09f4216 => generated 298 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:38.086816 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369f6b85-899c-468c-8533-eeec6c338515 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:38.087103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369f6b85-899c-468c-8533-eeec6c338515 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:38.087431 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369f6b85-899c-468c-8533-eeec6c338515 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:38.089008 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 75/152] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:38.092615 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369f6b85-899c-468c-8533-eeec6c338515 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:38.299722 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381d287f-8a8b-4cf7-a4ae-53d24e5b36ab None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:38.316332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369f6b85-899c-468c-8533-eeec6c338515 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:38.388718 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418ccc20-9ab8-4d21-882c-f21e209138e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:38.389361 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418ccc20-9ab8-4d21-882c-f21e209138e4 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:38.389641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418ccc20-9ab8-4d21-882c-f21e209138e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:38.389909 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418ccc20-9ab8-4d21-882c-f21e209138e4 None admin] PATH_INFO: `/v3/users/15dbb8e9805348eb9d1237a19e8e15f2` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:38.391756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-418ccc20-9ab8-4d21-882c-f21e209138e4 None admin] RBAC: Authorizing `identity:get_user(user_id=15dbb8e9805348eb9d1237a19e8e15f2)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:38.393206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-418ccc20-9ab8-4d21-882c-f21e209138e4 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:38.398009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 78/153] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:38 2022] GET /identity/v3/users/15dbb8e9805348eb9d1237a19e8e15f2 => generated 275 bytes in 351 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:38.404928 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78da17c4-9e73-42c9-9853-9bebc225a166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:38.405667 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78da17c4-9e73-42c9-9853-9bebc225a166 None admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:38.405947 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78da17c4-9e73-42c9-9853-9bebc225a166 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:38.406213 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78da17c4-9e73-42c9-9853-9bebc225a166 None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216/users/7a356c122f764e09b20f6bb7d3a6e028/roles/2c1b779eac624a728f7e590135c57450` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:38.407459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a121b082-97f2-46b9-ae8d-56e517a7f6f9 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:38.407735 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a121b082-97f2-46b9-ae8d-56e517a7f6f9 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:38.408005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a121b082-97f2-46b9-ae8d-56e517a7f6f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:38.411825 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78da17c4-9e73-42c9-9853-9bebc225a166 None admin] RBAC: Authorizing `identity:create_grant(project_id=feaf606da67b4ccf9411691df09f4216, user_id=7a356c122f764e09b20f6bb7d3a6e028, role_id=2c1b779eac624a728f7e590135c57450)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:38.413323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78da17c4-9e73-42c9-9853-9bebc225a166 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:38.443431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-a121b082-97f2-46b9-ae8d-56e517a7f6f9 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:38.444692 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a121b082-97f2-46b9-ae8d-56e517a7f6f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:38.470251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:38.513517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 76/154] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:42:38 2022] PUT /identity/v3/projects/feaf606da67b4ccf9411691df09f4216/users/7a356c122f764e09b20f6bb7d3a6e028/roles/2c1b779eac624a728f7e590135c57450 => generated 0 bytes in 421 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:42:38.514382 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78da17c4-9e73-42c9-9853-9bebc225a166 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:38.536694 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 79/155] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:38.566836 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:38.703177 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78da17c4-9e73-42c9-9853-9bebc225a166 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:38.744596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7bcd01-23bd-4b1a-9c62-42041a24adea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:38.745452 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7bcd01-23bd-4b1a-9c62-42041a24adea None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:38.746395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7bcd01-23bd-4b1a-9c62-42041a24adea None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:38.746802 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7bcd01-23bd-4b1a-9c62-42041a24adea None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:38.829395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7bcd01-23bd-4b1a-9c62-42041a24adea None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:38.831177 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7bcd01-23bd-4b1a-9c62-42041a24adea None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:38.846426 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:38.947018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 77/156] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:42:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:42:38.954238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a121b082-97f2-46b9-ae8d-56e517a7f6f9 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:39.068492 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a121b082-97f2-46b9-ae8d-56e517a7f6f9 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:39.113906 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4f7ede-ad09-4658-82b5-d515d589d9a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:39.114807 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4f7ede-ad09-4658-82b5-d515d589d9a5 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:39.115220 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4f7ede-ad09-4658-82b5-d515d589d9a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:39.115624 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4f7ede-ad09-4658-82b5-d515d589d9a5 None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:39.116896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4f7ede-ad09-4658-82b5-d515d589d9a5 None admin] RBAC: Authorizing `identity:get_project(project_id=feaf606da67b4ccf9411691df09f4216)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:39.118522 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4f7ede-ad09-4658-82b5-d515d589d9a5 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:39.120353 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 80/157] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:38 2022] GET /identity/v3/projects/feaf606da67b4ccf9411691df09f4216 => generated 298 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:39.127281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7bcd01-23bd-4b1a-9c62-42041a24adea None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:39.237133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7bcd01-23bd-4b1a-9c62-42041a24adea None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:39.262150 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:39.266807 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4f7ede-ad09-4658-82b5-d515d589d9a5 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:39.299748 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5606b279-1158-4906-82b6-ace5aeb6126d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:39.300375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5606b279-1158-4906-82b6-ace5aeb6126d None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:39.300658 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5606b279-1158-4906-82b6-ace5aeb6126d None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:39.300927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5606b279-1158-4906-82b6-ace5aeb6126d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:39.301669 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5606b279-1158-4906-82b6-ace5aeb6126d None admin] RBAC: Adding query filter params (role.id=e8367770d3124f2db1821b6af66bfeab, scope.project.id=feaf606da67b4ccf9411691df09f4216, user.id=15dbb8e9805348eb9d1237a19e8e15f2) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:39.301976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5606b279-1158-4906-82b6-ace5aeb6126d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:39.307502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5606b279-1158-4906-82b6-ace5aeb6126d None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:39.312232 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:39.315115 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 78/158] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:42:39 2022] GET /identity/v3/role_assignments?user.id=15dbb8e9805348eb9d1237a19e8e15f2&scope.project.id=feaf606da67b4ccf9411691df09f4216&role.id=e8367770d3124f2db1821b6af66bfeab => generated 608 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:39.422659 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4f7ede-ad09-4658-82b5-d515d589d9a5 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:39.469059 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4111b65-2dda-4388-9e14-6f3ed85d30d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:39.469699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4111b65-2dda-4388-9e14-6f3ed85d30d9 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:39.470032 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4111b65-2dda-4388-9e14-6f3ed85d30d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:39.470308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4111b65-2dda-4388-9e14-6f3ed85d30d9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:39.485819 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4111b65-2dda-4388-9e14-6f3ed85d30d9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:39.490960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-a4111b65-2dda-4388-9e14-6f3ed85d30d9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:42:39.490960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:39.490960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:39.490960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:39.490960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:39.490960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:39.490960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:39.490960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:39.490960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:39.490960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:39.490960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:39.490960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:39.490960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:39.490960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:39.490960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:39.490960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:39.491801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:39.491801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:39.491801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:39.491801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:39.491801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:39.491801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:39.491801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:39.491801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:39.491801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:39.491801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:39.491801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:39.491801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:39.491801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:39.491801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:39.491801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:39.491801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:39.492644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:39.492644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:39.492644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:39.492644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:39.492644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:39.492644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:39.492644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:39.492644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:39.492644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:39.492644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:39.492644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:42:39.492644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:42:39.492644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 81/159] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:42:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:42:39.498190 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5606b279-1158-4906-82b6-ace5aeb6126d None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:39.528165 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffb0ef9-1678-4365-b126-75d74d5d922e None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:39.528484 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffb0ef9-1678-4365-b126-75d74d5d922e None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:39.528714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ffb0ef9-1678-4365-b126-75d74d5d922e None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:39.530088 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 82/160] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:42:39.542483 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dc40f2-e5b4-4d5b-a6df-44e4669999b5 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:39.543544 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dc40f2-e5b4-4d5b-a6df-44e4669999b5 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:39.543814 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dc40f2-e5b4-4d5b-a6df-44e4669999b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:39.573686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-e9dc40f2-e5b4-4d5b-a6df-44e4669999b5 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:39.575477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e9dc40f2-e5b4-4d5b-a6df-44e4669999b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:39.684453 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5606b279-1158-4906-82b6-ace5aeb6126d None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:39.694970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 83/161] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 23 13:42:39.752714 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe74a128-4e6b-4c33-b0b1-2796db1299f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:39.753344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe74a128-4e6b-4c33-b0b1-2796db1299f3 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:39.753621 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe74a128-4e6b-4c33-b0b1-2796db1299f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:39.753888 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe74a128-4e6b-4c33-b0b1-2796db1299f3 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:39.754601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe74a128-4e6b-4c33-b0b1-2796db1299f3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:39.754900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe74a128-4e6b-4c33-b0b1-2796db1299f3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:39.756291 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe74a128-4e6b-4c33-b0b1-2796db1299f3 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:39.773764 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 79/162] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:39.780253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dc40f2-e5b4-4d5b-a6df-44e4669999b5 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:39.800993 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:39.898574 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dc40f2-e5b4-4d5b-a6df-44e4669999b5 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:39.942780 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2299de-82c1-43e6-b788-1c0999ebb0c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:39.943342 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2299de-82c1-43e6-b788-1c0999ebb0c7 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:39.943677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2299de-82c1-43e6-b788-1c0999ebb0c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:39.943953 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2299de-82c1-43e6-b788-1c0999ebb0c7 None admin] PATH_INFO: `/v3/users/2ea3af238714467ab48e451a5210c19f` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:39.945122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2299de-82c1-43e6-b788-1c0999ebb0c7 None admin] RBAC: Authorizing `identity:get_user(user_id=2ea3af238714467ab48e451a5210c19f)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:39.946586 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2299de-82c1-43e6-b788-1c0999ebb0c7 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:39.948173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 84/163] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:39 2022] GET /identity/v3/users/2ea3af238714467ab48e451a5210c19f => generated 247 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:39.959426 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe74a128-4e6b-4c33-b0b1-2796db1299f3 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:40.010242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33958bba-4f44-471a-8e9a-a750e6cd5b54 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:40.010783 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33958bba-4f44-471a-8e9a-a750e6cd5b54 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:40.010857 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33958bba-4f44-471a-8e9a-a750e6cd5b54 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:40.012495 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 85/164] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:40.019987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7e3d27-60eb-46f6-a7bb-609536c9890a None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:40.020567 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7e3d27-60eb-46f6-a7bb-609536c9890a None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:40.020846 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7e3d27-60eb-46f6-a7bb-609536c9890a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:40.047474 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:40.049098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-8d7e3d27-60eb-46f6-a7bb-609536c9890a None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:40.050301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8d7e3d27-60eb-46f6-a7bb-609536c9890a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:40.161946 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe74a128-4e6b-4c33-b0b1-2796db1299f3 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:40.178919 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 86/165] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 23 13:42:40.239994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afaf2e4-61b8-4e99-9eab-7df908ce863e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:40.240623 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afaf2e4-61b8-4e99-9eab-7df908ce863e None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:40.240901 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afaf2e4-61b8-4e99-9eab-7df908ce863e None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:40.241169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afaf2e4-61b8-4e99-9eab-7df908ce863e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:40.264665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7e3d27-60eb-46f6-a7bb-609536c9890a None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:40.286826 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:40.388645 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afaf2e4-61b8-4e99-9eab-7df908ce863e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:40.389505 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1afaf2e4-61b8-4e99-9eab-7df908ce863e None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:40.405763 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7e3d27-60eb-46f6-a7bb-609536c9890a None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:40.457836 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:40.466976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47352ab1-cc7c-4b49-b25a-77ee4b7ca29f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:40.467550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47352ab1-cc7c-4b49-b25a-77ee4b7ca29f None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:40.467829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47352ab1-cc7c-4b49-b25a-77ee4b7ca29f None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:40.468101 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47352ab1-cc7c-4b49-b25a-77ee4b7ca29f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:40.490778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47352ab1-cc7c-4b49-b25a-77ee4b7ca29f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:40.501296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-47352ab1-cc7c-4b49-b25a-77ee4b7ca29f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:42:40.501296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:40.501296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:40.501296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:40.501296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:40.501296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:40.501296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:40.501296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:40.501296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:40.501296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:40.501296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:40.501296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:40.501296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:40.501296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:40.501296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:40.501296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:40.502170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:40.502170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:40.502170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:40.502170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:40.502170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:40.502170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:40.502170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:40.502170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:40.502170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:40.502170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:40.502170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:40.502170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:40.502170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:40.502170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:40.502170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:40.502170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:40.503231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:40.503231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:40.503231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:40.503231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:40.503231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:40.503231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:40.503231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:40.503231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:40.503231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:40.503231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:40.503231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:42:40.503231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:42:40.503231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 87/166] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:42:40 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:42:40.515671 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47352ab1-cc7c-4b49-b25a-77ee4b7ca29f None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:40.581230 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 80/167] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:42:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 622 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:42:40.585865 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afaf2e4-61b8-4e99-9eab-7df908ce863e None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:40.636185 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:40.690400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47352ab1-cc7c-4b49-b25a-77ee4b7ca29f None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:40.755164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3cf94c-f8bc-4585-99d2-4a3e82ca1594 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:40.755632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3cf94c-f8bc-4585-99d2-4a3e82ca1594 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:40.755912 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3cf94c-f8bc-4585-99d2-4a3e82ca1594 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:40.756181 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3cf94c-f8bc-4585-99d2-4a3e82ca1594 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:40.756854 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3cf94c-f8bc-4585-99d2-4a3e82ca1594 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:40.757154 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3cf94c-f8bc-4585-99d2-4a3e82ca1594 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:40.758536 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3cf94c-f8bc-4585-99d2-4a3e82ca1594 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:40.760458 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afaf2e4-61b8-4e99-9eab-7df908ce863e None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:40.771618 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 88/168] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:42:40 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:40.772466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3cf94c-f8bc-4585-99d2-4a3e82ca1594 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:40.819267 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046a47a5-4a72-401c-8984-db3c0741a120 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:40.819967 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046a47a5-4a72-401c-8984-db3c0741a120 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:40.820247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046a47a5-4a72-401c-8984-db3c0741a120 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:40.820519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046a47a5-4a72-401c-8984-db3c0741a120 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:40.831954 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046a47a5-4a72-401c-8984-db3c0741a120 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:40.837026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-046a47a5-4a72-401c-8984-db3c0741a120 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:40.837026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:40.837026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:40.837026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:40.837026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:40.837026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:40.837026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:40.837026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:40.837026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:40.837026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:40.837026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:40.837026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:40.837026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:40.837026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:40.837026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:40.837026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:40.837988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:40.837988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:40.837988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:40.837988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:40.837988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:40.837988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:40.837988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:40.837988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:40.837988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:40.837988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:40.837988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:40.837988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:40.837988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:40.837988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:40.837988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:40.837988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:40.838793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:40.838793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:40.838793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:40.838793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:40.838793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:40.838793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:40.838793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:40.838793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:40.838793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:40.838793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:40.838793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:40.838793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:42:40.838793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 81/169] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:40 2022] GET /identity/v3/roles/member => generated 84 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:42:40.839495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046a47a5-4a72-401c-8984-db3c0741a120 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:40.857123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:40.950686 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046a47a5-4a72-401c-8984-db3c0741a120 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:40.987883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f938882e-09a6-4d3c-bb7f-3cd176dcaf7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:40.988517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f938882e-09a6-4d3c-bb7f-3cd176dcaf7a None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:40.988795 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f938882e-09a6-4d3c-bb7f-3cd176dcaf7a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:40.989066 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f938882e-09a6-4d3c-bb7f-3cd176dcaf7a None admin] PATH_INFO: `/v3/users/15dbb8e9805348eb9d1237a19e8e15f2` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:40.990195 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f938882e-09a6-4d3c-bb7f-3cd176dcaf7a None admin] RBAC: Authorizing `identity:get_user(user_id=15dbb8e9805348eb9d1237a19e8e15f2)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:40.991862 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f938882e-09a6-4d3c-bb7f-3cd176dcaf7a None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:40.993550 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 82/170] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:40 2022] GET /identity/v3/users/15dbb8e9805348eb9d1237a19e8e15f2 => generated 275 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:40.994953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241d85a7-275d-4f3d-b2cb-94ada2614cb0 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:40.995235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241d85a7-275d-4f3d-b2cb-94ada2614cb0 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:40.995508 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241d85a7-275d-4f3d-b2cb-94ada2614cb0 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:40.995606 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3cf94c-f8bc-4585-99d2-4a3e82ca1594 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:40.996746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 83/171] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:40.997682 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241d85a7-275d-4f3d-b2cb-94ada2614cb0 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:41.042125 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:41.050796 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0186a45-24d1-4eaf-a8d7-2ddcde0691a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:41.051437 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0186a45-24d1-4eaf-a8d7-2ddcde0691a7 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:41.051708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0186a45-24d1-4eaf-a8d7-2ddcde0691a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:41.051971 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0186a45-24d1-4eaf-a8d7-2ddcde0691a7 None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:41.053520 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0186a45-24d1-4eaf-a8d7-2ddcde0691a7 None admin] RBAC: Authorizing `identity:get_project(project_id=feaf606da67b4ccf9411691df09f4216)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:41.055383 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0186a45-24d1-4eaf-a8d7-2ddcde0691a7 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:41.057015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 89/172] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:40 2022] GET /identity/v3/projects/feaf606da67b4ccf9411691df09f4216 => generated 298 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:41.057767 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0186a45-24d1-4eaf-a8d7-2ddcde0691a7 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:41.134700 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:41.206007 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0186a45-24d1-4eaf-a8d7-2ddcde0691a7 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:41.207554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241d85a7-275d-4f3d-b2cb-94ada2614cb0 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:41.240578 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51bf02a4-8c03-4588-a783-f3a7d63a29fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:41.241207 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bf02a4-8c03-4588-a783-f3a7d63a29fc None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:41.241459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bf02a4-8c03-4588-a783-f3a7d63a29fc None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:41.241725 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51bf02a4-8c03-4588-a783-f3a7d63a29fc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:41.301190 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01f4a58-b75e-4740-bfeb-204e7da4b64e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:41.302168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01f4a58-b75e-4740-bfeb-204e7da4b64e None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:41.302694 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01f4a58-b75e-4740-bfeb-204e7da4b64e None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:41.303154 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01f4a58-b75e-4740-bfeb-204e7da4b64e None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:41.304009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01f4a58-b75e-4740-bfeb-204e7da4b64e None admin] RBAC: Adding query filter params (name=member) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:41.304495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01f4a58-b75e-4740-bfeb-204e7da4b64e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:41.306062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01f4a58-b75e-4740-bfeb-204e7da4b64e None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:41.325749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 84/173] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:42:40 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:41.327308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d17b31a-de2c-49c6-8c58-bd6de6b91f35 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:41.329887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d17b31a-de2c-49c6-8c58-bd6de6b91f35 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:41.330443 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d17b31a-de2c-49c6-8c58-bd6de6b91f35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:41.343488 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51bf02a4-8c03-4588-a783-f3a7d63a29fc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:41.347307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51bf02a4-8c03-4588-a783-f3a7d63a29fc None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:41.354536 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:41.371585 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-2d17b31a-de2c-49c6-8c58-bd6de6b91f35 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:41.372934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-2d17b31a-de2c-49c6-8c58-bd6de6b91f35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:41.511748 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 90/174] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:42:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 454 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:42:41.513450 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 85/175] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:41.514400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d17b31a-de2c-49c6-8c58-bd6de6b91f35 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:41.521641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51bf02a4-8c03-4588-a783-f3a7d63a29fc None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:41.627904 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:41.683529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d17b31a-de2c-49c6-8c58-bd6de6b91f35 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:41.702202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:41.706121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51bf02a4-8c03-4588-a783-f3a7d63a29fc None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:41.719902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe5028d-2468-4f23-a66c-ad8539779efd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:41.720537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe5028d-2468-4f23-a66c-ad8539779efd None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:41.720810 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe5028d-2468-4f23-a66c-ad8539779efd None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:41.721075 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe5028d-2468-4f23-a66c-ad8539779efd None admin] PATH_INFO: `/v3/users/15dbb8e9805348eb9d1237a19e8e15f2` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:41.722208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe5028d-2468-4f23-a66c-ad8539779efd None admin] RBAC: Authorizing `identity:get_user(user_id=15dbb8e9805348eb9d1237a19e8e15f2)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:41.723681 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe5028d-2468-4f23-a66c-ad8539779efd None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:41.725552 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 86/176] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:41 2022] GET /identity/v3/users/15dbb8e9805348eb9d1237a19e8e15f2 => generated 275 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:41.726087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe5028d-2468-4f23-a66c-ad8539779efd None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:41.746208 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95fb24a-eecd-4daf-a773-04615af3a169 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:41.746868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95fb24a-eecd-4daf-a773-04615af3a169 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:41.747152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95fb24a-eecd-4daf-a773-04615af3a169 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:41.747427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95fb24a-eecd-4daf-a773-04615af3a169 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:41.748327 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95fb24a-eecd-4daf-a773-04615af3a169 None admin] RBAC: Adding query filter params (role.id=727f0fa598e54b9a929d2cf6277c875e, scope.project.id=feaf606da67b4ccf9411691df09f4216, user.id=2ea3af238714467ab48e451a5210c19f) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:41.748733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95fb24a-eecd-4daf-a773-04615af3a169 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:41.751079 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95fb24a-eecd-4daf-a773-04615af3a169 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:41.757388 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 91/177] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:42:41 2022] GET /identity/v3/role_assignments?user.id=2ea3af238714467ab48e451a5210c19f&scope.project.id=feaf606da67b4ccf9411691df09f4216&role.id=727f0fa598e54b9a929d2cf6277c875e => generated 608 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:41.758591 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46068ddb-9275-45a5-a292-fac7cf0451dc None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:41.758911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46068ddb-9275-45a5-a292-fac7cf0451dc None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:41.759188 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46068ddb-9275-45a5-a292-fac7cf0451dc None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:41.760550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 92/178] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:41.761173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46068ddb-9275-45a5-a292-fac7cf0451dc None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:41.940989 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe5028d-2468-4f23-a66c-ad8539779efd None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:41.958185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:41.963649 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46068ddb-9275-45a5-a292-fac7cf0451dc None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:41.971502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27d575c-9fd8-487e-bb5f-ae449ccee107 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:41.971502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27d575c-9fd8-487e-bb5f-ae449ccee107 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:41.971502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27d575c-9fd8-487e-bb5f-ae449ccee107 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:41.971801 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27d575c-9fd8-487e-bb5f-ae449ccee107 None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:41.973074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27d575c-9fd8-487e-bb5f-ae449ccee107 None admin] RBAC: Authorizing `identity:get_project(project_id=feaf606da67b4ccf9411691df09f4216)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:41.975501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27d575c-9fd8-487e-bb5f-ae449ccee107 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:41.978009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 87/179] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:41 2022] GET /identity/v3/projects/feaf606da67b4ccf9411691df09f4216 => generated 298 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:41.982487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043fd260-9f63-4038-8d7f-ec69cbeb4182 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:41.983030 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043fd260-9f63-4038-8d7f-ec69cbeb4182 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:41.983473 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043fd260-9f63-4038-8d7f-ec69cbeb4182 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:42.006866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-043fd260-9f63-4038-8d7f-ec69cbeb4182 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:42.008391 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-043fd260-9f63-4038-8d7f-ec69cbeb4182 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:42.014244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba37710-2e46-4169-9689-c9b3ca389396 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:42.014901 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba37710-2e46-4169-9689-c9b3ca389396 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:42.015169 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba37710-2e46-4169-9689-c9b3ca389396 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:42.015434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba37710-2e46-4169-9689-c9b3ca389396 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:42.051174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:42.093315 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 88/180] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:42.094422 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00c4e53-fabf-4070-9924-4bd7b7be1b77 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:42.094705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00c4e53-fabf-4070-9924-4bd7b7be1b77 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:42.094971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00c4e53-fabf-4070-9924-4bd7b7be1b77 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:42.100368 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 89/181] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:42.100368 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00c4e53-fabf-4070-9924-4bd7b7be1b77 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:42.105705 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba37710-2e46-4169-9689-c9b3ca389396 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:42.107657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ba37710-2e46-4169-9689-c9b3ca389396 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:42.223972 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00c4e53-fabf-4070-9924-4bd7b7be1b77 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:42.224312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 93/182] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:42:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 463 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:42:42.229071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba37710-2e46-4169-9689-c9b3ca389396 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:42.260051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:42.267292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d56ff2-2c15-470b-9b42-f511957d4469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:42.267944 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d56ff2-2c15-470b-9b42-f511957d4469 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:42.268223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d56ff2-2c15-470b-9b42-f511957d4469 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:42.268572 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d56ff2-2c15-470b-9b42-f511957d4469 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:42.279183 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d56ff2-2c15-470b-9b42-f511957d4469 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:42.284051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-50d56ff2-2c15-470b-9b42-f511957d4469 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:42:42.284051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:42.284051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:42.284051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:42.284051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:42.284051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:42.284051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:42.284051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:42.284051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:42.284051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:42.284051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:42.284051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:42.284051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:42.284051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:42.284051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:42.284051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:42.285000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:42.285000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:42.285000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:42.285000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:42.285000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:42.285000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:42.285000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:42.285000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:42.285000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:42.285000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:42.285000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:42.285000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:42.285000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:42.285000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:42.285000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:42.285000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:42.287298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:42.287298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:42.287298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:42.287298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:42.287298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:42.287298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:42.287298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:42.287298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:42.287298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:42.287298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:42.287298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:42:42.287298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:42:42.287298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 90/183] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:42:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:42.287983 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7cacee-addf-42ef-8395-17c6c1eede7e None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:42.287983 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7cacee-addf-42ef-8395-17c6c1eede7e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:42.288307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7cacee-addf-42ef-8395-17c6c1eede7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:42.306255 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-5e7cacee-addf-42ef-8395-17c6c1eede7e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:42.307477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-5e7cacee-addf-42ef-8395-17c6c1eede7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:42.316875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:42.336075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba37710-2e46-4169-9689-c9b3ca389396 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:42.373062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ea9a81-457a-450b-9bab-6ffe44de4e3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:42.373668 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ea9a81-457a-450b-9bab-6ffe44de4e3b None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:42.373971 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ea9a81-457a-450b-9bab-6ffe44de4e3b None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:42.374540 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ea9a81-457a-450b-9bab-6ffe44de4e3b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:42.375269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ea9a81-457a-450b-9bab-6ffe44de4e3b None admin] RBAC: Adding query filter params (role.id=5ef6f90ed37e44adafc451b081a501d1, scope.project.id=feaf606da67b4ccf9411691df09f4216, user.id=15dbb8e9805348eb9d1237a19e8e15f2) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:42.375572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ea9a81-457a-450b-9bab-6ffe44de4e3b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:42.377006 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ea9a81-457a-450b-9bab-6ffe44de4e3b None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:42.377106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 91/184] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:42.378293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16423fa1-1aa6-4161-880b-d73572c8fcac None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:42.378608 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16423fa1-1aa6-4161-880b-d73572c8fcac None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:42.378908 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16423fa1-1aa6-4161-880b-d73572c8fcac None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:42.382369 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 92/185] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:42.382369 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16423fa1-1aa6-4161-880b-d73572c8fcac None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:42.384698 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 94/186] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:42:42 2022] GET /identity/v3/role_assignments?user.id=15dbb8e9805348eb9d1237a19e8e15f2&scope.project.id=feaf606da67b4ccf9411691df09f4216&role.id=5ef6f90ed37e44adafc451b081a501d1 => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:42.385422 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ea9a81-457a-450b-9bab-6ffe44de4e3b None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:42.502622 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16423fa1-1aa6-4161-880b-d73572c8fcac None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:42.510657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ea9a81-457a-450b-9bab-6ffe44de4e3b None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:42.519643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:42.543875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdae3ed3-adf1-4ca0-9899-47e093917157 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:42.544193 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:42.545013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdae3ed3-adf1-4ca0-9899-47e093917157 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:42.545511 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdae3ed3-adf1-4ca0-9899-47e093917157 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:42.546009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdae3ed3-adf1-4ca0-9899-47e093917157 None admin] PATH_INFO: `/v3/projects/4652340ee06c4fc387b2b690cab9ee8d` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:42.547470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdae3ed3-adf1-4ca0-9899-47e093917157 None admin] RBAC: Authorizing `identity:get_project(project_id=4652340ee06c4fc387b2b690cab9ee8d)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:42.549244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdae3ed3-adf1-4ca0-9899-47e093917157 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:42.555747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71442449-602b-428b-a3a2-8cc2a62e33be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:42.555747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71442449-602b-428b-a3a2-8cc2a62e33be None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:42.555747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71442449-602b-428b-a3a2-8cc2a62e33be None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:42.555747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71442449-602b-428b-a3a2-8cc2a62e33be None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:42.556146 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71442449-602b-428b-a3a2-8cc2a62e33be None admin] RBAC: Adding query filter params (name=admin) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:42.556146 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71442449-602b-428b-a3a2-8cc2a62e33be None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:42.556484 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71442449-602b-428b-a3a2-8cc2a62e33be None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:42.557377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 93/187] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:42 2022] GET /identity/v3/projects/4652340ee06c4fc387b2b690cab9ee8d => generated 312 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:42.559304 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdae3ed3-adf1-4ca0-9899-47e093917157 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:42.567762 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 95/188] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:42.574391 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d458d0-789c-4752-ae6d-e5e83b30d622 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:42.574391 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d458d0-789c-4752-ae6d-e5e83b30d622 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:42.574391 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3d458d0-789c-4752-ae6d-e5e83b30d622 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:42.592657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-f3d458d0-789c-4752-ae6d-e5e83b30d622 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:42.593849 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f3d458d0-789c-4752-ae6d-e5e83b30d622 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:42.686437 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 96/189] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:42.687318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d458d0-789c-4752-ae6d-e5e83b30d622 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:42.699175 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdae3ed3-adf1-4ca0-9899-47e093917157 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:42.737798 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98238af9-f4f3-4e1d-a24d-04006d93d165 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:42.738461 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98238af9-f4f3-4e1d-a24d-04006d93d165 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:42.738736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98238af9-f4f3-4e1d-a24d-04006d93d165 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:42.739000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98238af9-f4f3-4e1d-a24d-04006d93d165 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:42.749140 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98238af9-f4f3-4e1d-a24d-04006d93d165 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:42.754665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-98238af9-f4f3-4e1d-a24d-04006d93d165 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:42:42.754665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:42.754665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:42.754665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:42.754665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:42.754665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:42.754665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:42.754665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:42.754665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:42.754665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:42.754665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:42.754665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:42.754665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:42.754665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:42.754665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:42.754665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:42.755500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:42.755500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:42.755500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:42.755500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:42.755500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:42.755500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:42.755500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:42.755500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:42.755500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:42.755500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:42.755500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:42.755500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:42.755500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:42.755500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:42.755500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:42.755500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:42.756289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:42.756289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:42.756289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:42.756289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:42.756289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:42.756289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:42.756289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:42.756289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:42.756289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:42.756289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:42.756289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:42:42.756289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:42:42.756289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 94/190] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:42:42 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:42.757536 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98238af9-f4f3-4e1d-a24d-04006d93d165 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:42.797956 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3d458d0-789c-4752-ae6d-e5e83b30d622 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:42.801198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:42.819847 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:42.834647 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57901870-6ebd-48b7-978b-734531746e59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:42.835251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57901870-6ebd-48b7-978b-734531746e59 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:42.835533 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57901870-6ebd-48b7-978b-734531746e59 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:42.835799 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57901870-6ebd-48b7-978b-734531746e59 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:42.836514 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57901870-6ebd-48b7-978b-734531746e59 None admin] RBAC: Adding query filter params (role.id=e8367770d3124f2db1821b6af66bfeab, scope.project.id=4652340ee06c4fc387b2b690cab9ee8d, user.id=15dbb8e9805348eb9d1237a19e8e15f2) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:42.836821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57901870-6ebd-48b7-978b-734531746e59 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:42.838245 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57901870-6ebd-48b7-978b-734531746e59 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:42.845080 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 97/191] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:42:42 2022] GET /identity/v3/role_assignments?user.id=15dbb8e9805348eb9d1237a19e8e15f2&scope.project.id=4652340ee06c4fc387b2b690cab9ee8d&role.id=e8367770d3124f2db1821b6af66bfeab => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:42.851515 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57901870-6ebd-48b7-978b-734531746e59 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:42.872160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98238af9-f4f3-4e1d-a24d-04006d93d165 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:42.924079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e8dc13-f142-4492-9e2f-5e7122d4d8e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:42.924553 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e8dc13-f142-4492-9e2f-5e7122d4d8e0 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:42.924907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e8dc13-f142-4492-9e2f-5e7122d4d8e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:42.925291 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e8dc13-f142-4492-9e2f-5e7122d4d8e0 None admin] PATH_INFO: `/v3/users/1ad6dce8df8349ea9c810da3a6b26864` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:42.944154 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e8dc13-f142-4492-9e2f-5e7122d4d8e0 None admin] RBAC: Authorizing `identity:get_user(user_id=1ad6dce8df8349ea9c810da3a6b26864)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:42.946197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e8dc13-f142-4492-9e2f-5e7122d4d8e0 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:42.947869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 95/192] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:42 2022] GET /identity/v3/users/1ad6dce8df8349ea9c810da3a6b26864 => generated 283 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:42.949418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e8dc13-f142-4492-9e2f-5e7122d4d8e0 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:42.978689 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57901870-6ebd-48b7-978b-734531746e59 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:43.015928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c04648-e6f3-4ae2-93aa-0e3d34e3b1bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:43.016489 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c04648-e6f3-4ae2-93aa-0e3d34e3b1bc None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:43.016773 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c04648-e6f3-4ae2-93aa-0e3d34e3b1bc None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:43.017043 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c04648-e6f3-4ae2-93aa-0e3d34e3b1bc None admin] PATH_INFO: `/v3/domains/default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:43.018216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c04648-e6f3-4ae2-93aa-0e3d34e3b1bc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:43.019720 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c04648-e6f3-4ae2-93aa-0e3d34e3b1bc None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:43.021332 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 98/193] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:42:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:43.022112 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c04648-e6f3-4ae2-93aa-0e3d34e3b1bc None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:43.078726 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:43.082149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e8dc13-f142-4492-9e2f-5e7122d4d8e0 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:43.090957 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:43.122421 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b016a353-626d-4d69-b04e-63d080317fa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:43.123029 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b016a353-626d-4d69-b04e-63d080317fa4 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:43.123308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b016a353-626d-4d69-b04e-63d080317fa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:43.123576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b016a353-626d-4d69-b04e-63d080317fa4 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:43.124505 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b016a353-626d-4d69-b04e-63d080317fa4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:43.124807 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b016a353-626d-4d69-b04e-63d080317fa4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:43.126200 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b016a353-626d-4d69-b04e-63d080317fa4 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:43.135303 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 96/194] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:42:42 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:43.135303 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b016a353-626d-4d69-b04e-63d080317fa4 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:43.144595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c04648-e6f3-4ae2-93aa-0e3d34e3b1bc None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:43.178879 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f45d59-4923-4fb6-bc28-5c3e6dd2eba6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:43.179486 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f45d59-4923-4fb6-bc28-5c3e6dd2eba6 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:43.179765 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f45d59-4923-4fb6-bc28-5c3e6dd2eba6 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:43.180034 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f45d59-4923-4fb6-bc28-5c3e6dd2eba6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:43.237655 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b016a353-626d-4d69-b04e-63d080317fa4 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:43.251842 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f45d59-4923-4fb6-bc28-5c3e6dd2eba6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:43.253314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f45d59-4923-4fb6-bc28-5c3e6dd2eba6 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:43.271752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7dcbea-f519-4a9d-848f-6bb38e9f6f7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:43.272306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7dcbea-f519-4a9d-848f-6bb38e9f6f7f None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:43.272582 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7dcbea-f519-4a9d-848f-6bb38e9f6f7f None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:43.272845 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7dcbea-f519-4a9d-848f-6bb38e9f6f7f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:43.282957 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7dcbea-f519-4a9d-848f-6bb38e9f6f7f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:43.288101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-ec7dcbea-f519-4a9d-848f-6bb38e9f6f7f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 23 13:42:43.288101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:43.288101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:43.288101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:43.288101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:43.288101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:43.288101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:43.288101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:43.288101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:43.288101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:43.288101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:43.288101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:43.288101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:43.288101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:43.288101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:43.288101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:43.289119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:43.289119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:43.289119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:43.289119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:43.289119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:43.289119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:43.289119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:43.289119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:43.289119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:43.289119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:43.289119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:43.289119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:43.289119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:43.289119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:43.289119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:43.289119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:43.289887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:43.289887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:43.289887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:43.289887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:43.289887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:43.289887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:43.289887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:43.289887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:43.289887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:43.289887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:43.289887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 23 13:42:43.289887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:42:43.289887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 97/195] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:43 2022] GET /identity/v3/roles/reader => generated 84 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:43.289887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7dcbea-f519-4a9d-848f-6bb38e9f6f7f None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:43.306501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:43.314955 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:43.354217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 99/196] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:42:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:42:43.355109 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f45d59-4923-4fb6-bc28-5c3e6dd2eba6 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:43.398850 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7dcbea-f519-4a9d-848f-6bb38e9f6f7f None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:43.440649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a8f8a0-832a-4353-a304-5e7edb9c4d7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:43.441269 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a8f8a0-832a-4353-a304-5e7edb9c4d7d None admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:43.441547 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a8f8a0-832a-4353-a304-5e7edb9c4d7d None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:43.441820 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a8f8a0-832a-4353-a304-5e7edb9c4d7d None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:43.443314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a8f8a0-832a-4353-a304-5e7edb9c4d7d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:43.445319 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a8f8a0-832a-4353-a304-5e7edb9c4d7d None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:43.457393 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f45d59-4923-4fb6-bc28-5c3e6dd2eba6 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:43.490614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f5fae7-3d09-4ac1-8b26-b4e6fbcdfccb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:43.491238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f5fae7-3d09-4ac1-8b26-b4e6fbcdfccb None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:43.491517 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f5fae7-3d09-4ac1-8b26-b4e6fbcdfccb None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:43.491799 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f5fae7-3d09-4ac1-8b26-b4e6fbcdfccb None admin] PATH_INFO: `/v3/users/1ad6dce8df8349ea9c810da3a6b26864` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:43.492910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f5fae7-3d09-4ac1-8b26-b4e6fbcdfccb None admin] RBAC: Authorizing `identity:get_user(user_id=1ad6dce8df8349ea9c810da3a6b26864)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:43.494353 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f5fae7-3d09-4ac1-8b26-b4e6fbcdfccb None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:43.500697 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 100/197] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:43 2022] GET /identity/v3/users/1ad6dce8df8349ea9c810da3a6b26864 => generated 283 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:43.500697 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f5fae7-3d09-4ac1-8b26-b4e6fbcdfccb None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:43.517608 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:43.562075 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 98/198] 10.222.0.62 () {60 vars in 1132 bytes} [Fri Sep 23 13:42:43 2022] POST /identity/v3/users => generated 297 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:42:43.562834 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a8f8a0-832a-4353-a304-5e7edb9c4d7d None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:43.606482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f5fae7-3d09-4ac1-8b26-b4e6fbcdfccb None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:43.656165 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78289c1a-bbc2-4bef-90b0-d75709b9e456 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:43.656970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78289c1a-bbc2-4bef-90b0-d75709b9e456 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:43.657354 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78289c1a-bbc2-4bef-90b0-d75709b9e456 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:43.658496 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78289c1a-bbc2-4bef-90b0-d75709b9e456 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:43.658496 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78289c1a-bbc2-4bef-90b0-d75709b9e456 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:43.658904 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78289c1a-bbc2-4bef-90b0-d75709b9e456 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:43.660434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78289c1a-bbc2-4bef-90b0-d75709b9e456 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:43.671026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 101/199] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:42:43 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:43.671427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78289c1a-bbc2-4bef-90b0-d75709b9e456 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:43.678749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:43.707310 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a8f8a0-832a-4353-a304-5e7edb9c4d7d None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:43.756227 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd7bce2-03cf-44da-b274-782c62ef24a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:43.756910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd7bce2-03cf-44da-b274-782c62ef24a7 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:43.757194 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd7bce2-03cf-44da-b274-782c62ef24a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:43.757465 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd7bce2-03cf-44da-b274-782c62ef24a7 None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:43.758706 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd7bce2-03cf-44da-b274-782c62ef24a7 None admin] RBAC: Authorizing `identity:get_project(project_id=8d4ba5b452824beab548b1646270d12c)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:43.760652 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd7bce2-03cf-44da-b274-782c62ef24a7 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:43.763536 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 99/200] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:43 2022] GET /identity/v3/projects/8d4ba5b452824beab548b1646270d12c => generated 302 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:43.767239 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd7bce2-03cf-44da-b274-782c62ef24a7 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:43.778457 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:43.804762 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78289c1a-bbc2-4bef-90b0-d75709b9e456 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:43.857773 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fa6a1f-6f60-4fa0-9a3a-d275d93aed52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:43.859060 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fa6a1f-6f60-4fa0-9a3a-d275d93aed52 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:43.859060 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fa6a1f-6f60-4fa0-9a3a-d275d93aed52 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:43.859060 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fa6a1f-6f60-4fa0-9a3a-d275d93aed52 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:43.889651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd7bce2-03cf-44da-b274-782c62ef24a7 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:43.935108 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a0dcbc-1b64-4201-97a2-70b6d6910bbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:43.935848 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a0dcbc-1b64-4201-97a2-70b6d6910bbf None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:43.936203 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a0dcbc-1b64-4201-97a2-70b6d6910bbf None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:43.936551 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a0dcbc-1b64-4201-97a2-70b6d6910bbf None admin] PATH_INFO: `/v3/users/7a356c122f764e09b20f6bb7d3a6e028` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:43.937799 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a0dcbc-1b64-4201-97a2-70b6d6910bbf None admin] RBAC: Authorizing `identity:get_user(user_id=7a356c122f764e09b20f6bb7d3a6e028)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:43.940320 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a0dcbc-1b64-4201-97a2-70b6d6910bbf None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:43.942056 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 100/201] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:43 2022] GET /identity/v3/users/7a356c122f764e09b20f6bb7d3a6e028 => generated 289 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:43.942951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a0dcbc-1b64-4201-97a2-70b6d6910bbf None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:43.962138 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:43.966312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fa6a1f-6f60-4fa0-9a3a-d275d93aed52 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:43.968545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fa6a1f-6f60-4fa0-9a3a-d275d93aed52 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:44.067737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a0dcbc-1b64-4201-97a2-70b6d6910bbf None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:44.081969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:44.105914 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 102/202] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:42:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:42:44.108814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a955fe-1ac5-469a-9378-24a4d8c8a5ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:44.109428 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a955fe-1ac5-469a-9378-24a4d8c8a5ab None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:44.109772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a955fe-1ac5-469a-9378-24a4d8c8a5ab None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:44.110160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a955fe-1ac5-469a-9378-24a4d8c8a5ab None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:44.110965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a955fe-1ac5-469a-9378-24a4d8c8a5ab None admin] RBAC: Adding query filter params (role.id=727f0fa598e54b9a929d2cf6277c875e, scope.project.id=8d4ba5b452824beab548b1646270d12c, user.id=1ad6dce8df8349ea9c810da3a6b26864) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:44.111346 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a955fe-1ac5-469a-9378-24a4d8c8a5ab None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:44.111750 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fa6a1f-6f60-4fa0-9a3a-d275d93aed52 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:44.113687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a955fe-1ac5-469a-9378-24a4d8c8a5ab None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:44.128511 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 101/203] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:42:43 2022] GET /identity/v3/role_assignments?user.id=1ad6dce8df8349ea9c810da3a6b26864&scope.project.id=8d4ba5b452824beab548b1646270d12c&role.id=727f0fa598e54b9a929d2cf6277c875e => generated 259 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:44.135641 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a955fe-1ac5-469a-9378-24a4d8c8a5ab None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:44.261569 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fa6a1f-6f60-4fa0-9a3a-d275d93aed52 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:44.305890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487234e5-abb2-4e9c-82f5-60da66f6c5f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:44.306959 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487234e5-abb2-4e9c-82f5-60da66f6c5f6 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:44.307670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487234e5-abb2-4e9c-82f5-60da66f6c5f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:44.308710 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487234e5-abb2-4e9c-82f5-60da66f6c5f6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:44.378156 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:44.394735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:44.404914 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487234e5-abb2-4e9c-82f5-60da66f6c5f6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:44.408086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487234e5-abb2-4e9c-82f5-60da66f6c5f6 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:44.434092 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a955fe-1ac5-469a-9378-24a4d8c8a5ab None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:44.503683 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bf8870-ace0-48f5-8d8e-c4ec4c55cd50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:44.504337 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bf8870-ace0-48f5-8d8e-c4ec4c55cd50 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:44.504619 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bf8870-ace0-48f5-8d8e-c4ec4c55cd50 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:44.504890 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bf8870-ace0-48f5-8d8e-c4ec4c55cd50 None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:44.509248 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bf8870-ace0-48f5-8d8e-c4ec4c55cd50 None admin] RBAC: Authorizing `identity:get_project(project_id=8d4ba5b452824beab548b1646270d12c)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:44.510832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bf8870-ace0-48f5-8d8e-c4ec4c55cd50 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:44.513178 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 102/204] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:44 2022] GET /identity/v3/projects/8d4ba5b452824beab548b1646270d12c => generated 302 bytes in 384 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:44.520044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bf8870-ace0-48f5-8d8e-c4ec4c55cd50 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:44.538980 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 103/205] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:42:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:42:44.547427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487234e5-abb2-4e9c-82f5-60da66f6c5f6 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:44.719411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:44.733202 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bf8870-ace0-48f5-8d8e-c4ec4c55cd50 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:44.753534 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487234e5-abb2-4e9c-82f5-60da66f6c5f6 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:44.805753 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f891b9-6e4b-40f9-a5e7-5d07de2407e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:44.806389 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f891b9-6e4b-40f9-a5e7-5d07de2407e6 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:44.806678 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f891b9-6e4b-40f9-a5e7-5d07de2407e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:44.806940 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f891b9-6e4b-40f9-a5e7-5d07de2407e6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:44.808645 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f891b9-6e4b-40f9-a5e7-5d07de2407e6 None admin] RBAC: Adding query filter params (role.id=5ef6f90ed37e44adafc451b081a501d1, scope.project.id=8d4ba5b452824beab548b1646270d12c, user.id=1ad6dce8df8349ea9c810da3a6b26864) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:44.809233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f891b9-6e4b-40f9-a5e7-5d07de2407e6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:44.810718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4f891b9-6e4b-40f9-a5e7-5d07de2407e6 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:44.821550 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 103/206] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:42:44 2022] GET /identity/v3/role_assignments?user.id=1ad6dce8df8349ea9c810da3a6b26864&scope.project.id=8d4ba5b452824beab548b1646270d12c&role.id=5ef6f90ed37e44adafc451b081a501d1 => generated 259 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:44.835428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbe98ee-cfe2-4e46-8983-69fc9af5803a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:44.836117 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbe98ee-cfe2-4e46-8983-69fc9af5803a None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:44.836403 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbe98ee-cfe2-4e46-8983-69fc9af5803a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:44.836675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbe98ee-cfe2-4e46-8983-69fc9af5803a None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:44.838040 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbe98ee-cfe2-4e46-8983-69fc9af5803a None admin] RBAC: Authorizing `identity:get_project(project_id=feaf606da67b4ccf9411691df09f4216)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:44.839644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbe98ee-cfe2-4e46-8983-69fc9af5803a None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:44.841412 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 104/207] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:44 2022] GET /identity/v3/projects/feaf606da67b4ccf9411691df09f4216 => generated 298 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:44.847859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f891b9-6e4b-40f9-a5e7-5d07de2407e6 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:44.871513 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:44.983313 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f891b9-6e4b-40f9-a5e7-5d07de2407e6 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:45.020947 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb21b49-8403-485f-9478-aa5f3bf9fb17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:45.021553 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb21b49-8403-485f-9478-aa5f3bf9fb17 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:45.021827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb21b49-8403-485f-9478-aa5f3bf9fb17 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:45.022094 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb21b49-8403-485f-9478-aa5f3bf9fb17 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:45.022853 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb21b49-8403-485f-9478-aa5f3bf9fb17 None admin] RBAC: Adding query filter params (role.id=2c1b779eac624a728f7e590135c57450, scope.project.id=feaf606da67b4ccf9411691df09f4216, user.id=7a356c122f764e09b20f6bb7d3a6e028) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:45.023154 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb21b49-8403-485f-9478-aa5f3bf9fb17 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:45.024589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb21b49-8403-485f-9478-aa5f3bf9fb17 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:45.031501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 104/208] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:42:44 2022] GET /identity/v3/role_assignments?user.id=7a356c122f764e09b20f6bb7d3a6e028&scope.project.id=feaf606da67b4ccf9411691df09f4216&role.id=2c1b779eac624a728f7e590135c57450 => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:46.145808 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aae198-5b94-4f76-9c4d-8cbc6586edf3 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:46.146317 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aae198-5b94-4f76-9c4d-8cbc6586edf3 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:46.146577 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aae198-5b94-4f76-9c4d-8cbc6586edf3 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:46.148156 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 105/209] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:46.154346 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07162c4a-44f9-4e20-a42b-eea205f172fe None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:46.155188 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07162c4a-44f9-4e20-a42b-eea205f172fe None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:46.155676 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07162c4a-44f9-4e20-a42b-eea205f172fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:46.174933 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-07162c4a-44f9-4e20-a42b-eea205f172fe None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:46.176437 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-07162c4a-44f9-4e20-a42b-eea205f172fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:46.230577 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65c7598-496e-42dd-ab81-c1eb03e5b71d None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:46.231203 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65c7598-496e-42dd-ab81-c1eb03e5b71d None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:46.231703 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65c7598-496e-42dd-ab81-c1eb03e5b71d None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:46.233147 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 106/210] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:42:46.238404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe80b91-c0c2-45f7-bfaa-f5560a7d9616 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:46.238808 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe80b91-c0c2-45f7-bfaa-f5560a7d9616 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:46.239134 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe80b91-c0c2-45f7-bfaa-f5560a7d9616 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:46.254453 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 105/211] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 23 13:42:46.258444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-cfe80b91-c0c2-45f7-bfaa-f5560a7d9616 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:46.259714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-cfe80b91-c0c2-45f7-bfaa-f5560a7d9616 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:46.331382 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 107/212] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 23 13:42:46.520692 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07162c4a-44f9-4e20-a42b-eea205f172fe None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:46.524159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:46.607084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe80b91-c0c2-45f7-bfaa-f5560a7d9616 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:46.632035 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07162c4a-44f9-4e20-a42b-eea205f172fe None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:46.675261 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735a5a90-662a-4c6b-b6ce-89a9a77354e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:46.676156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735a5a90-662a-4c6b-b6ce-89a9a77354e2 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:46.676441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735a5a90-662a-4c6b-b6ce-89a9a77354e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:46.676710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735a5a90-662a-4c6b-b6ce-89a9a77354e2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:46.690396 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735a5a90-662a-4c6b-b6ce-89a9a77354e2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:46.697073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-735a5a90-662a-4c6b-b6ce-89a9a77354e2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:46.697073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:46.697073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:46.697073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:46.697073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:46.697073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:46.697073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:46.697073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:46.697073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:46.697073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:46.697073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:46.697073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:46.697073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:46.697073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:46.697073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:46.697073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:46.697951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:46.697951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:46.697951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:46.697951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:46.697951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:46.697951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:46.697951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:46.697951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:46.697951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:46.697951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:46.697951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:46.697951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:46.697951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:46.697951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:46.697951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:46.697951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:46.699738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:46.699738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:46.699738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:46.699738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:46.699738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:46.699738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:46.699738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:46.699738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:46.699738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:46.699738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:46.699738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:46.699738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:42:46.699738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 106/213] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:46 2022] GET /identity/v3/roles/member => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:46.699738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd908f1a-fdb0-4d44-b2a0-d7d9f7c30a15 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:46.700461 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd908f1a-fdb0-4d44-b2a0-d7d9f7c30a15 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:46.700461 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd908f1a-fdb0-4d44-b2a0-d7d9f7c30a15 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:46.701300 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 107/214] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:46.703801 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd908f1a-fdb0-4d44-b2a0-d7d9f7c30a15 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:46.705632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:46.726215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe80b91-c0c2-45f7-bfaa-f5560a7d9616 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:46.764126 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3dddaf-4d40-43f7-9ce4-1b66df9e4c3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:46.764757 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3dddaf-4d40-43f7-9ce4-1b66df9e4c3d None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:46.765035 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3dddaf-4d40-43f7-9ce4-1b66df9e4c3d None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:46.765303 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3dddaf-4d40-43f7-9ce4-1b66df9e4c3d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:46.766434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb3dddaf-4d40-43f7-9ce4-1b66df9e4c3d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:46.767905 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb3dddaf-4d40-43f7-9ce4-1b66df9e4c3d None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:46.769836 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 108/215] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:42:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:46.771224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5332e94c-179c-474e-8538-64076d1f2347 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:46.771446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5332e94c-179c-474e-8538-64076d1f2347 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:46.771717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5332e94c-179c-474e-8538-64076d1f2347 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:46.775364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:46.791103 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-5332e94c-179c-474e-8538-64076d1f2347 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:46.792853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-5332e94c-179c-474e-8538-64076d1f2347 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:46.816397 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd908f1a-fdb0-4d44-b2a0-d7d9f7c30a15 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:46.851617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5e13ec-0d2d-46a1-960e-5e6de56f797f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:46.852220 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5e13ec-0d2d-46a1-960e-5e6de56f797f None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:46.852496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5e13ec-0d2d-46a1-960e-5e6de56f797f None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:46.852762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5e13ec-0d2d-46a1-960e-5e6de56f797f None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:46.853425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5e13ec-0d2d-46a1-960e-5e6de56f797f None admin] RBAC: Adding query filter params (name=member) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:46.853721 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5e13ec-0d2d-46a1-960e-5e6de56f797f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:46.855790 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5e13ec-0d2d-46a1-960e-5e6de56f797f None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:46.860998 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 109/216] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:46.866608 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5332e94c-179c-474e-8538-64076d1f2347 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:46.869964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 108/217] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:42:46 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:46.876379 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5e13ec-0d2d-46a1-960e-5e6de56f797f None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:46.966941 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:46.994752 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5332e94c-179c-474e-8538-64076d1f2347 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:47.015837 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5e13ec-0d2d-46a1-960e-5e6de56f797f None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:47.043781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f22a4c1-11bd-46a8-8148-fd06d4410528 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:47.044758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f22a4c1-11bd-46a8-8148-fd06d4410528 None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:47.045237 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f22a4c1-11bd-46a8-8148-fd06d4410528 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:47.046505 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f22a4c1-11bd-46a8-8148-fd06d4410528 None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:47.047726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f22a4c1-11bd-46a8-8148-fd06d4410528 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:47.050393 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f22a4c1-11bd-46a8-8148-fd06d4410528 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:47.052636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:47.063648 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecbe821-9b31-45ce-ab03-2c0c884292dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:47.064290 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecbe821-9b31-45ce-ab03-2c0c884292dc None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:47.065018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecbe821-9b31-45ce-ab03-2c0c884292dc None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:47.066388 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ecbe821-9b31-45ce-ab03-2c0c884292dc None admin] PATH_INFO: `/v3/users/ba2b60a965074d58b885dbc37040042a` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:47.080697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ecbe821-9b31-45ce-ab03-2c0c884292dc None admin] RBAC: Authorizing `identity:get_user(user_id=ba2b60a965074d58b885dbc37040042a)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:47.082192 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ecbe821-9b31-45ce-ab03-2c0c884292dc None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:47.084099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 109/218] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:46 2022] GET /identity/v3/users/ba2b60a965074d58b885dbc37040042a => generated 297 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:47.089813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecbe821-9b31-45ce-ab03-2c0c884292dc None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:47.150094 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 110/219] 10.222.0.62 () {60 vars in 1132 bytes} [Fri Sep 23 13:42:46 2022] POST /identity/v3/users => generated 297 bytes in 289 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:42:47.152138 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f22a4c1-11bd-46a8-8148-fd06d4410528 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:47.211603 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ecbe821-9b31-45ce-ab03-2c0c884292dc None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:47.255654 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c055a2-afed-47a4-af72-69c67d678e8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:47.256330 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c055a2-afed-47a4-af72-69c67d678e8f None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:47.256604 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c055a2-afed-47a4-af72-69c67d678e8f None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:47.256983 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c055a2-afed-47a4-af72-69c67d678e8f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:47.282080 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f22a4c1-11bd-46a8-8148-fd06d4410528 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:47.318667 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:47.330724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed5bec0-10ae-44a1-b0c8-bafd8a122b2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:47.334499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed5bec0-10ae-44a1-b0c8-bafd8a122b2b None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:47.334499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed5bec0-10ae-44a1-b0c8-bafd8a122b2b None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:47.334499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed5bec0-10ae-44a1-b0c8-bafd8a122b2b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:47.334499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:47.336649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c055a2-afed-47a4-af72-69c67d678e8f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:47.337997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c055a2-afed-47a4-af72-69c67d678e8f None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:47.347677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed5bec0-10ae-44a1-b0c8-bafd8a122b2b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:47.353222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-bed5bec0-10ae-44a1-b0c8-bafd8a122b2b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:42:47.353222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:47.353222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:47.353222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:47.353222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:47.353222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:47.353222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:47.353222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:47.353222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:47.353222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:47.353222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:47.353222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:47.353222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:47.353222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:47.353222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:47.353222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:47.354071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:47.354071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:47.354071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:47.354071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:47.354071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:47.354071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:47.354071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:47.354071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:47.354071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:47.354071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:47.354071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:47.354071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:47.354071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:47.354071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:47.354071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:47.354071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:47.354925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:47.354925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:47.354925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:47.354925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:47.354925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:47.354925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:47.354925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:47.354925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:47.354925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:47.354925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:47.354925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:42:47.354925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:42:47.354925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 111/220] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:42:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:47.356118 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417bad74-5ee2-4474-a473-c443dd8a8f1d None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:47.356118 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417bad74-5ee2-4474-a473-c443dd8a8f1d None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:47.356277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417bad74-5ee2-4474-a473-c443dd8a8f1d None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:47.357498 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 112/221] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:47.360209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417bad74-5ee2-4474-a473-c443dd8a8f1d None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:47.467506 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 110/222] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:42:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 23 13:42:47.469278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb6cbf7-4fcd-43d3-ac0e-917a57eefae2 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:47.469278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb6cbf7-4fcd-43d3-ac0e-917a57eefae2 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:47.469278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb6cbf7-4fcd-43d3-ac0e-917a57eefae2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:47.494580 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-3eb6cbf7-4fcd-43d3-ac0e-917a57eefae2 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:47.496865 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-3eb6cbf7-4fcd-43d3-ac0e-917a57eefae2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:47.508222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417bad74-5ee2-4474-a473-c443dd8a8f1d None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:47.552912 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229f524b-2231-4489-8f4e-cff8068b627f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:47.553627 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229f524b-2231-4489-8f4e-cff8068b627f None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:47.553906 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229f524b-2231-4489-8f4e-cff8068b627f None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:47.554175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229f524b-2231-4489-8f4e-cff8068b627f None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:47.556013 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229f524b-2231-4489-8f4e-cff8068b627f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:47.556013 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229f524b-2231-4489-8f4e-cff8068b627f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:47.556627 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229f524b-2231-4489-8f4e-cff8068b627f None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:47.567386 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 113/223] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:47.568212 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229f524b-2231-4489-8f4e-cff8068b627f None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:47.579473 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 111/224] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:47.580493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb6cbf7-4fcd-43d3-ac0e-917a57eefae2 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:47.597597 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:47.638136 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:47.724680 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229f524b-2231-4489-8f4e-cff8068b627f None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:47.759734 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb6cbf7-4fcd-43d3-ac0e-917a57eefae2 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:47.795982 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c69110-cdfd-4690-b742-4c15cb6d087b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:47.796618 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c69110-cdfd-4690-b742-4c15cb6d087b None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:47.796895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c69110-cdfd-4690-b742-4c15cb6d087b None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:47.797158 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c69110-cdfd-4690-b742-4c15cb6d087b None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:47.799031 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c69110-cdfd-4690-b742-4c15cb6d087b None admin] RBAC: Authorizing `identity:get_project(project_id=8d4ba5b452824beab548b1646270d12c)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:47.800470 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c69110-cdfd-4690-b742-4c15cb6d087b None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:47.805229 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 114/225] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:47 2022] GET /identity/v3/projects/8d4ba5b452824beab548b1646270d12c => generated 302 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:47.807891 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c69110-cdfd-4690-b742-4c15cb6d087b None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:47.842471 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627b0b5d-6dfe-419f-b317-0f439c3596dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:47.843095 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627b0b5d-6dfe-419f-b317-0f439c3596dd None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:47.843490 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627b0b5d-6dfe-419f-b317-0f439c3596dd None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:47.843756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627b0b5d-6dfe-419f-b317-0f439c3596dd None admin] PATH_INFO: `/v3/users/1ad6dce8df8349ea9c810da3a6b26864` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:47.851064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627b0b5d-6dfe-419f-b317-0f439c3596dd None admin] RBAC: Authorizing `identity:get_user(user_id=1ad6dce8df8349ea9c810da3a6b26864)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:47.852524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627b0b5d-6dfe-419f-b317-0f439c3596dd None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:47.854308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 112/226] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:47 2022] GET /identity/v3/users/1ad6dce8df8349ea9c810da3a6b26864 => generated 283 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:47.859891 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627b0b5d-6dfe-419f-b317-0f439c3596dd None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:47.928812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c69110-cdfd-4690-b742-4c15cb6d087b None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:47.955063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:47.967181 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030bc2cb-74c3-4c65-b23c-64a1c4b14901 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:47.967793 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030bc2cb-74c3-4c65-b23c-64a1c4b14901 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:47.976170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030bc2cb-74c3-4c65-b23c-64a1c4b14901 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:47.976170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030bc2cb-74c3-4c65-b23c-64a1c4b14901 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:47.976170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030bc2cb-74c3-4c65-b23c-64a1c4b14901 None admin] RBAC: Adding query filter params (role.id=e8367770d3124f2db1821b6af66bfeab, scope.project.id=8d4ba5b452824beab548b1646270d12c, user.id=ba2b60a965074d58b885dbc37040042a) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:47.976170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030bc2cb-74c3-4c65-b23c-64a1c4b14901 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:47.976170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030bc2cb-74c3-4c65-b23c-64a1c4b14901 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:47.976492 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:47.979014 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 115/227] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:42:47 2022] GET /identity/v3/role_assignments?user.id=ba2b60a965074d58b885dbc37040042a&scope.project.id=8d4ba5b452824beab548b1646270d12c&role.id=e8367770d3124f2db1821b6af66bfeab => generated 259 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:47.982211 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030bc2cb-74c3-4c65-b23c-64a1c4b14901 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:47.988571 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627b0b5d-6dfe-419f-b317-0f439c3596dd None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:48.072332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e7db7e-529c-4cad-a149-12a25c09a602 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:48.075413 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e7db7e-529c-4cad-a149-12a25c09a602 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:48.076131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e7db7e-529c-4cad-a149-12a25c09a602 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:48.078696 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e7db7e-529c-4cad-a149-12a25c09a602 None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:48.080437 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e7db7e-529c-4cad-a149-12a25c09a602 None admin] RBAC: Authorizing `identity:get_project(project_id=8d4ba5b452824beab548b1646270d12c)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:48.083033 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e7db7e-529c-4cad-a149-12a25c09a602 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:48.086408 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 113/228] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:47 2022] GET /identity/v3/projects/8d4ba5b452824beab548b1646270d12c => generated 302 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:48.094448 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e7db7e-529c-4cad-a149-12a25c09a602 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:48.173012 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030bc2cb-74c3-4c65-b23c-64a1c4b14901 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:48.225429 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e7db7e-529c-4cad-a149-12a25c09a602 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:48.234311 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b2e48b-38c0-4525-a0b1-8099a5aada16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:48.235026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b2e48b-38c0-4525-a0b1-8099a5aada16 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:48.235306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b2e48b-38c0-4525-a0b1-8099a5aada16 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:48.235608 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b2e48b-38c0-4525-a0b1-8099a5aada16 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:48.246428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b2e48b-38c0-4525-a0b1-8099a5aada16 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:48.252026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-13b2e48b-38c0-4525-a0b1-8099a5aada16 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:42:48.252026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:48.252026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:48.252026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:48.252026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:48.252026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:48.252026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:48.252026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:48.252026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:48.252026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:48.252026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:48.252026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:48.252026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:48.252026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:48.252026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:48.252026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:48.252886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:48.252886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:48.252886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:48.252886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:48.252886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:48.252886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:48.252886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:48.252886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:48.252886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:48.252886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:48.252886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:48.252886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:48.252886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:48.252886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:48.252886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:48.252886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:48.253679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:48.253679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:48.253679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:48.253679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:48.253679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:48.253679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:48.253679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:48.253679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:48.253679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:48.253679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:48.253679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:42:48.253679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:42:48.253679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 116/229] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:42:47 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:48.258925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b2e48b-38c0-4525-a0b1-8099a5aada16 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:48.270608 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cb11b6-6d8a-41b6-a097-d48ec7912830 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:48.271368 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cb11b6-6d8a-41b6-a097-d48ec7912830 None admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:48.271756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cb11b6-6d8a-41b6-a097-d48ec7912830 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:48.272033 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cb11b6-6d8a-41b6-a097-d48ec7912830 None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c/users/1ad6dce8df8349ea9c810da3a6b26864/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:48.274866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46cb11b6-6d8a-41b6-a097-d48ec7912830 None admin] RBAC: Authorizing `identity:create_grant(project_id=8d4ba5b452824beab548b1646270d12c, user_id=1ad6dce8df8349ea9c810da3a6b26864, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:48.276965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46cb11b6-6d8a-41b6-a097-d48ec7912830 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:48.310933 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:48.367011 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 114/230] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:42:48 2022] PUT /identity/v3/projects/8d4ba5b452824beab548b1646270d12c/users/1ad6dce8df8349ea9c810da3a6b26864/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:42:48.372398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b2e48b-38c0-4525-a0b1-8099a5aada16 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:48.420012 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb85bbbc-a7bd-4d33-96ec-67dbbb6aa0dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:48.420636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb85bbbc-a7bd-4d33-96ec-67dbbb6aa0dc None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:48.420912 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb85bbbc-a7bd-4d33-96ec-67dbbb6aa0dc None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:48.421180 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb85bbbc-a7bd-4d33-96ec-67dbbb6aa0dc None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:48.421913 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb85bbbc-a7bd-4d33-96ec-67dbbb6aa0dc None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:48.422219 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb85bbbc-a7bd-4d33-96ec-67dbbb6aa0dc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:48.423667 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb85bbbc-a7bd-4d33-96ec-67dbbb6aa0dc None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:48.441663 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 117/231] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:42:48 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:48.447221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cb11b6-6d8a-41b6-a097-d48ec7912830 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:48.470581 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:48.597528 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cb11b6-6d8a-41b6-a097-d48ec7912830 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:48.641241 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f58c17-921c-445e-aac4-076f7ef6fc00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:48.642268 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f58c17-921c-445e-aac4-076f7ef6fc00 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:48.642777 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f58c17-921c-445e-aac4-076f7ef6fc00 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:48.643245 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f58c17-921c-445e-aac4-076f7ef6fc00 None admin] PATH_INFO: `/v3/users/1ad6dce8df8349ea9c810da3a6b26864` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:48.644623 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f58c17-921c-445e-aac4-076f7ef6fc00 None admin] RBAC: Authorizing `identity:get_user(user_id=1ad6dce8df8349ea9c810da3a6b26864)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:48.646288 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f58c17-921c-445e-aac4-076f7ef6fc00 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:48.647921 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 115/232] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:48 2022] GET /identity/v3/users/1ad6dce8df8349ea9c810da3a6b26864 => generated 283 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:48.654356 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb85bbbc-a7bd-4d33-96ec-67dbbb6aa0dc None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:48.768257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb85bbbc-a7bd-4d33-96ec-67dbbb6aa0dc None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:48.783843 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:48.813895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939f5960-bc6a-48c5-bc84-1eb16f73a700 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:48.814571 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939f5960-bc6a-48c5-bc84-1eb16f73a700 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:48.814855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939f5960-bc6a-48c5-bc84-1eb16f73a700 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:48.815128 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939f5960-bc6a-48c5-bc84-1eb16f73a700 None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:48.816286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939f5960-bc6a-48c5-bc84-1eb16f73a700 None admin] RBAC: Authorizing `identity:get_project(project_id=8d4ba5b452824beab548b1646270d12c)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:48.817732 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939f5960-bc6a-48c5-bc84-1eb16f73a700 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:48.819562 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 118/233] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:48 2022] GET /identity/v3/projects/8d4ba5b452824beab548b1646270d12c => generated 302 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:48.825874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f58c17-921c-445e-aac4-076f7ef6fc00 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:48.946013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:48.968887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f58c17-921c-445e-aac4-076f7ef6fc00 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:49.005281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9e5b1d-23f4-437f-9adf-347d15937a23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:49.005948 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9e5b1d-23f4-437f-9adf-347d15937a23 None admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:49.006225 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9e5b1d-23f4-437f-9adf-347d15937a23 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:49.006525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9e5b1d-23f4-437f-9adf-347d15937a23 None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c/users/1ad6dce8df8349ea9c810da3a6b26864/roles/5ef6f90ed37e44adafc451b081a501d1` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:49.008672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9e5b1d-23f4-437f-9adf-347d15937a23 None admin] RBAC: Authorizing `identity:create_grant(project_id=8d4ba5b452824beab548b1646270d12c, user_id=1ad6dce8df8349ea9c810da3a6b26864, role_id=5ef6f90ed37e44adafc451b081a501d1)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:49.010108 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9e5b1d-23f4-437f-9adf-347d15937a23 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:49.128275 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 116/234] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:42:48 2022] PUT /identity/v3/projects/8d4ba5b452824beab548b1646270d12c/users/1ad6dce8df8349ea9c810da3a6b26864/roles/5ef6f90ed37e44adafc451b081a501d1 => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:42:49.848651 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf9cd49-e7ae-4fbc-a3f9-41092fa5567a None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:49.849187 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf9cd49-e7ae-4fbc-a3f9-41092fa5567a None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:49.849187 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf9cd49-e7ae-4fbc-a3f9-41092fa5567a None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:49.850832 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 119/235] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:49.856229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06bb7a2-1db0-4e08-a6ab-1eadd660d32e None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:49.856633 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06bb7a2-1db0-4e08-a6ab-1eadd660d32e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:49.856798 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06bb7a2-1db0-4e08-a6ab-1eadd660d32e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:49.879550 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-d06bb7a2-1db0-4e08-a6ab-1eadd660d32e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:49.880739 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d06bb7a2-1db0-4e08-a6ab-1eadd660d32e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:49.968593 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 117/236] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:49.999683 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ae9ebc-b217-4986-ab0b-24f847949779 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:49.999994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ae9ebc-b217-4986-ab0b-24f847949779 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:50.000233 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ae9ebc-b217-4986-ab0b-24f847949779 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:50.001800 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 120/237] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:50.006923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1258d4-2aca-4023-8ccd-ae3ae3632db2 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:50.007208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1258d4-2aca-4023-8ccd-ae3ae3632db2 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:50.007482 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1258d4-2aca-4023-8ccd-ae3ae3632db2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:50.034565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-8f1258d4-2aca-4023-8ccd-ae3ae3632db2 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:50.035762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8f1258d4-2aca-4023-8ccd-ae3ae3632db2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:50.117520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:50.123468 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 118/238] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:50.402890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ae9ebc-b217-4986-ab0b-24f847949779 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:50.547943 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1258d4-2aca-4023-8ccd-ae3ae3632db2 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:50.555087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ae9ebc-b217-4986-ab0b-24f847949779 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:50.599003 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249ba870-8888-46e8-b937-339bca95b3eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:50.600144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249ba870-8888-46e8-b937-339bca95b3eb None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:50.601418 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249ba870-8888-46e8-b937-339bca95b3eb None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:50.602541 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249ba870-8888-46e8-b937-339bca95b3eb None admin] PATH_INFO: `/v3/domains/default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:50.609957 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249ba870-8888-46e8-b937-339bca95b3eb None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:50.612330 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249ba870-8888-46e8-b937-339bca95b3eb None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:50.616309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 121/239] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:42:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:50.627676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249ba870-8888-46e8-b937-339bca95b3eb None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:50.699448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:50.753375 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249ba870-8888-46e8-b937-339bca95b3eb None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:50.764375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1258d4-2aca-4023-8ccd-ae3ae3632db2 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:50.794421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88cff55-059a-4ddd-99c3-c73e7e3b5446 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:50.795385 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88cff55-059a-4ddd-99c3-c73e7e3b5446 None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:50.795968 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88cff55-059a-4ddd-99c3-c73e7e3b5446 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:50.796460 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88cff55-059a-4ddd-99c3-c73e7e3b5446 None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:50.797461 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88cff55-059a-4ddd-99c3-c73e7e3b5446 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:50.799282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88cff55-059a-4ddd-99c3-c73e7e3b5446 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:50.812795 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29319ccb-2790-49f2-be27-df9ad27046a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:50.813739 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29319ccb-2790-49f2-be27-df9ad27046a2 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:50.814219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29319ccb-2790-49f2-be27-df9ad27046a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:50.814708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29319ccb-2790-49f2-be27-df9ad27046a2 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:50.829736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29319ccb-2790-49f2-be27-df9ad27046a2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:50.835583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-29319ccb-2790-49f2-be27-df9ad27046a2 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 23 13:42:50.835583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:50.835583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:50.835583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:50.835583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:50.835583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:50.835583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:50.835583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:50.835583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:50.835583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:50.835583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:50.835583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:50.835583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:50.835583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:50.835583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:50.835583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:50.843919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:50.843919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:50.843919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:50.843919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:50.843919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:50.843919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:50.843919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:50.843919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:50.843919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:50.843919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:50.843919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:50.843919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:50.843919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:50.843919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:50.843919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:50.843919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:50.844704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:50.844704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:50.844704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:50.844704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:50.844704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:50.844704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:50.844704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:50.844704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:50.844704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:50.844704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:50.844704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 23 13:42:50.844704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:42:50.844704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 119/240] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:50 2022] GET /identity/v3/roles/reader => generated 84 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:42:50.844704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29319ccb-2790-49f2-be27-df9ad27046a2 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:50.892098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 122/241] 10.222.0.62 () {60 vars in 1132 bytes} [Fri Sep 23 13:42:50 2022] POST /identity/v3/users => generated 293 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:42:50.946264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:50.986453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340a5d6a-3056-4144-97cd-84a4318772a3 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:50.987111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340a5d6a-3056-4144-97cd-84a4318772a3 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:50.987614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340a5d6a-3056-4144-97cd-84a4318772a3 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:50.989174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 123/242] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:42:50.993657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29319ccb-2790-49f2-be27-df9ad27046a2 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:50.995615 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dab2869-465d-455c-a74d-9c16ed9fc8e6 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:50.996847 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dab2869-465d-455c-a74d-9c16ed9fc8e6 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:50.997457 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dab2869-465d-455c-a74d-9c16ed9fc8e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:51.018539 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-8dab2869-465d-455c-a74d-9c16ed9fc8e6 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:51.020320 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8dab2869-465d-455c-a74d-9c16ed9fc8e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:51.055601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1716be4-62db-4c26-92e4-d89108bda950 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:51.056289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1716be4-62db-4c26-92e4-d89108bda950 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:51.056570 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1716be4-62db-4c26-92e4-d89108bda950 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:51.056840 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1716be4-62db-4c26-92e4-d89108bda950 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:51.058144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1716be4-62db-4c26-92e4-d89108bda950 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:51.058565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1716be4-62db-4c26-92e4-d89108bda950 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:51.060279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1716be4-62db-4c26-92e4-d89108bda950 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:51.072835 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 120/243] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:42:50 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:51.074358 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efee8d3b-2ada-4ca5-811a-2bfd2f0c04af None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:51.074698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efee8d3b-2ada-4ca5-811a-2bfd2f0c04af None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:51.074969 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efee8d3b-2ada-4ca5-811a-2bfd2f0c04af None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:51.076347 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 121/244] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:51.081196 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efee8d3b-2ada-4ca5-811a-2bfd2f0c04af None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:51.106378 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 124/245] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 23 13:42:51.118302 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4e56b4-0e17-482d-9b95-1fea8dac23a4 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:51.118302 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4e56b4-0e17-482d-9b95-1fea8dac23a4 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:51.118302 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4e56b4-0e17-482d-9b95-1fea8dac23a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:51.155308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-4a4e56b4-0e17-482d-9b95-1fea8dac23a4 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:51.156547 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-4a4e56b4-0e17-482d-9b95-1fea8dac23a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:51.188442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:51.234358 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efee8d3b-2ada-4ca5-811a-2bfd2f0c04af None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:51.248177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 125/246] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:51.258886 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:51.273319 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55f9ab1-8f70-408b-9efa-e3e00eafda61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:51.277467 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55f9ab1-8f70-408b-9efa-e3e00eafda61 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:51.277467 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55f9ab1-8f70-408b-9efa-e3e00eafda61 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:51.277467 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55f9ab1-8f70-408b-9efa-e3e00eafda61 None admin] PATH_INFO: `/v3/users/0a0dead8d66c4e8395b7a5f3780e7930` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:51.290153 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55f9ab1-8f70-408b-9efa-e3e00eafda61 None admin] RBAC: Authorizing `identity:get_user(user_id=0a0dead8d66c4e8395b7a5f3780e7930)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:51.292455 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a55f9ab1-8f70-408b-9efa-e3e00eafda61 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:51.294730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 122/247] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:51 2022] GET /identity/v3/users/0a0dead8d66c4e8395b7a5f3780e7930 => generated 297 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:51.300683 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4e56b4-0e17-482d-9b95-1fea8dac23a4 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:51.408301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4e56b4-0e17-482d-9b95-1fea8dac23a4 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:51.426919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55f9ab1-8f70-408b-9efa-e3e00eafda61 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:51.447052 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac33869-2a56-4b96-ada0-c9e950842ca5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:51.449040 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac33869-2a56-4b96-ada0-c9e950842ca5 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:51.449561 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac33869-2a56-4b96-ada0-c9e950842ca5 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:51.450001 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac33869-2a56-4b96-ada0-c9e950842ca5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:51.483667 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:51.535030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac33869-2a56-4b96-ada0-c9e950842ca5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:51.536747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac33869-2a56-4b96-ada0-c9e950842ca5 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:51.569885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55f9ab1-8f70-408b-9efa-e3e00eafda61 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:51.606349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a978441f-d8bc-4856-9d32-29427ca639e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:51.606988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a978441f-d8bc-4856-9d32-29427ca639e7 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:51.607262 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a978441f-d8bc-4856-9d32-29427ca639e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:51.607516 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a978441f-d8bc-4856-9d32-29427ca639e7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:51.618185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a978441f-d8bc-4856-9d32-29427ca639e7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:51.623191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-a978441f-d8bc-4856-9d32-29427ca639e7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:51.623191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:51.623191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:51.623191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:51.623191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:51.623191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:51.623191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:51.623191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:51.623191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:51.623191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:51.623191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:51.623191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:51.623191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:51.623191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:51.623191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:51.623191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:51.624357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:51.624357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:51.624357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:51.624357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:51.624357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:51.624357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:51.624357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:51.624357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:51.624357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:51.624357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:51.624357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:51.624357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:51.624357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:51.624357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:51.624357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:51.624357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:51.625134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:51.625134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:51.625134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:51.625134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:51.625134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:51.625134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:51.625134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:51.625134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:51.625134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:51.625134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:51.625134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:51.625134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:42:51.625134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 123/248] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:42:51.625781 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a978441f-d8bc-4856-9d32-29427ca639e7 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:51.667950 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 126/249] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:42:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:42:51.668827 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac33869-2a56-4b96-ada0-c9e950842ca5 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:51.683532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:51.747063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:51.772366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a978441f-d8bc-4856-9d32-29427ca639e7 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:51.805661 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac33869-2a56-4b96-ada0-c9e950842ca5 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:51.823669 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d2ff30-bd46-4ccc-a0ee-c627829c6463 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:51.824347 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d2ff30-bd46-4ccc-a0ee-c627829c6463 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:51.824631 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d2ff30-bd46-4ccc-a0ee-c627829c6463 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:51.825095 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d2ff30-bd46-4ccc-a0ee-c627829c6463 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:51.841024 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d2ff30-bd46-4ccc-a0ee-c627829c6463 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:51.846039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-d6d2ff30-bd46-4ccc-a0ee-c627829c6463 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:42:51.846039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:51.846039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:51.846039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:51.846039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:51.846039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:51.846039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:51.846039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:51.846039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:51.846039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:51.846039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:51.846039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:51.846039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:51.846039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:51.846039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:51.846039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:51.846929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:51.846929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:51.846929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:51.846929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:51.846929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:51.846929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:51.846929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:51.846929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:51.846929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:51.846929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:51.846929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:51.846929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:51.846929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:51.846929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:51.846929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:51.846929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:51.847698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:51.847698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:51.847698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:51.847698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:51.847698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:51.847698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:51.847698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:51.847698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:51.847698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:51.847698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:51.847698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:42:51.847698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:42:51.847698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 124/250] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:42:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:51.848983 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d2ff30-bd46-4ccc-a0ee-c627829c6463 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:51.868962 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e79ccac-c209-4e02-8071-297728f1f133 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:51.869578 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e79ccac-c209-4e02-8071-297728f1f133 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:51.869923 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e79ccac-c209-4e02-8071-297728f1f133 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:51.870706 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e79ccac-c209-4e02-8071-297728f1f133 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:51.871384 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e79ccac-c209-4e02-8071-297728f1f133 None admin] RBAC: Adding query filter params (name=member) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:51.871753 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e79ccac-c209-4e02-8071-297728f1f133 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:51.873468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e79ccac-c209-4e02-8071-297728f1f133 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:51.889146 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 127/251] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:42:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:51.889815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e79ccac-c209-4e02-8071-297728f1f133 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:52.001280 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:52.005587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d2ff30-bd46-4ccc-a0ee-c627829c6463 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:52.014838 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e79ccac-c209-4e02-8071-297728f1f133 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:52.043362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5cf667-6824-426e-bd30-535b4090373a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:52.044593 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5cf667-6824-426e-bd30-535b4090373a None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:52.045232 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5cf667-6824-426e-bd30-535b4090373a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:52.045987 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5cf667-6824-426e-bd30-535b4090373a None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:52.047747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5cf667-6824-426e-bd30-535b4090373a None admin] RBAC: Authorizing `identity:get_project(project_id=8d4ba5b452824beab548b1646270d12c)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:52.049594 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5cf667-6824-426e-bd30-535b4090373a None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:52.050512 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714c3958-283e-49a2-a81e-8837d9562743 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:52.051128 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714c3958-283e-49a2-a81e-8837d9562743 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:52.051412 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714c3958-283e-49a2-a81e-8837d9562743 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:52.051684 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714c3958-283e-49a2-a81e-8837d9562743 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:52.052412 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714c3958-283e-49a2-a81e-8837d9562743 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:52.052511 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 125/252] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:51 2022] GET /identity/v3/projects/8d4ba5b452824beab548b1646270d12c => generated 302 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:52.052724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714c3958-283e-49a2-a81e-8837d9562743 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:52.057930 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714c3958-283e-49a2-a81e-8837d9562743 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:52.059389 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5cf667-6824-426e-bd30-535b4090373a None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:52.065433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 128/253] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:52.066752 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7346ee97-7dd7-4bc0-b3ce-626099688cd9 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:52.067043 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7346ee97-7dd7-4bc0-b3ce-626099688cd9 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:52.067317 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7346ee97-7dd7-4bc0-b3ce-626099688cd9 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:52.068786 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 129/254] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:52.069500 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7346ee97-7dd7-4bc0-b3ce-626099688cd9 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:52.103586 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:52.188271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7346ee97-7dd7-4bc0-b3ce-626099688cd9 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:52.196981 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5cf667-6824-426e-bd30-535b4090373a None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:52.233889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb871b4-f18c-4d3c-8429-8a1b4aaf7edf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:52.234678 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb871b4-f18c-4d3c-8429-8a1b4aaf7edf None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:52.235210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb871b4-f18c-4d3c-8429-8a1b4aaf7edf None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:52.235573 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb871b4-f18c-4d3c-8429-8a1b4aaf7edf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:52.236806 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb871b4-f18c-4d3c-8429-8a1b4aaf7edf None admin] RBAC: Adding query filter params (role.id=2c1b779eac624a728f7e590135c57450, scope.project.id=8d4ba5b452824beab548b1646270d12c, user.id=0a0dead8d66c4e8395b7a5f3780e7930) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:52.237437 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb871b4-f18c-4d3c-8429-8a1b4aaf7edf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:52.238978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb871b4-f18c-4d3c-8429-8a1b4aaf7edf None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:52.244395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f3d655-a2e5-49d9-89fd-c62197380545 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:52.245790 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f3d655-a2e5-49d9-89fd-c62197380545 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:52.246530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 130/255] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:42:52 2022] GET /identity/v3/role_assignments?user.id=0a0dead8d66c4e8395b7a5f3780e7930&scope.project.id=8d4ba5b452824beab548b1646270d12c&role.id=2c1b779eac624a728f7e590135c57450 => generated 259 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:52.246746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f3d655-a2e5-49d9-89fd-c62197380545 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:52.247181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f3d655-a2e5-49d9-89fd-c62197380545 None admin] PATH_INFO: `/v3/users/ba2b60a965074d58b885dbc37040042a` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:52.248795 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f3d655-a2e5-49d9-89fd-c62197380545 None admin] RBAC: Authorizing `identity:get_user(user_id=ba2b60a965074d58b885dbc37040042a)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:52.250507 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f3d655-a2e5-49d9-89fd-c62197380545 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:52.252391 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 126/256] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:52 2022] GET /identity/v3/users/ba2b60a965074d58b885dbc37040042a => generated 297 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:52.257734 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f3d655-a2e5-49d9-89fd-c62197380545 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:52.257870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a254a8d7-cd59-46d4-a7c7-aa3ba4e7b8f4 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:52.257870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a254a8d7-cd59-46d4-a7c7-aa3ba4e7b8f4 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:52.257870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a254a8d7-cd59-46d4-a7c7-aa3ba4e7b8f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:52.287816 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-a254a8d7-cd59-46d4-a7c7-aa3ba4e7b8f4 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:52.289356 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a254a8d7-cd59-46d4-a7c7-aa3ba4e7b8f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:52.326431 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:52.368871 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:52.392641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 131/257] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:52.393947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a254a8d7-cd59-46d4-a7c7-aa3ba4e7b8f4 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:52.490224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f3d655-a2e5-49d9-89fd-c62197380545 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:52.531708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2713f5-1a09-43c4-8b98-7e17537ce277 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:52.532349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2713f5-1a09-43c4-8b98-7e17537ce277 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:52.532630 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2713f5-1a09-43c4-8b98-7e17537ce277 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:52.532902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2713f5-1a09-43c4-8b98-7e17537ce277 None admin] PATH_INFO: `/v3/users/1ad6dce8df8349ea9c810da3a6b26864` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:52.534950 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2713f5-1a09-43c4-8b98-7e17537ce277 None admin] RBAC: Authorizing `identity:get_user(user_id=1ad6dce8df8349ea9c810da3a6b26864)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:52.536403 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2713f5-1a09-43c4-8b98-7e17537ce277 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:52.538315 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 127/258] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:52 2022] GET /identity/v3/users/1ad6dce8df8349ea9c810da3a6b26864 => generated 283 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:52.544200 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2713f5-1a09-43c4-8b98-7e17537ce277 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:52.592987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a254a8d7-cd59-46d4-a7c7-aa3ba4e7b8f4 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:52.656069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6927e26d-e934-4f23-8370-54c1651d7b4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:52.656714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6927e26d-e934-4f23-8370-54c1651d7b4c None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:52.656991 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6927e26d-e934-4f23-8370-54c1651d7b4c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:52.657263 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6927e26d-e934-4f23-8370-54c1651d7b4c None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:52.659923 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6927e26d-e934-4f23-8370-54c1651d7b4c None admin] RBAC: Authorizing `identity:get_project(project_id=8d4ba5b452824beab548b1646270d12c)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:52.661369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6927e26d-e934-4f23-8370-54c1651d7b4c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:52.663570 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 132/259] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:52 2022] GET /identity/v3/projects/8d4ba5b452824beab548b1646270d12c => generated 302 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:52.671515 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6927e26d-e934-4f23-8370-54c1651d7b4c None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:52.727462 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2713f5-1a09-43c4-8b98-7e17537ce277 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:52.729870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:52.765052 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:52.787838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6038a16f-3ad3-440e-8705-294c77f80160 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:52.788484 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6038a16f-3ad3-440e-8705-294c77f80160 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:52.788778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6038a16f-3ad3-440e-8705-294c77f80160 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:52.789037 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6038a16f-3ad3-440e-8705-294c77f80160 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:52.864553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6927e26d-e934-4f23-8370-54c1651d7b4c None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:52.890220 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6038a16f-3ad3-440e-8705-294c77f80160 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:52.891718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6038a16f-3ad3-440e-8705-294c77f80160 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:52.932699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ef3c5f-0206-4e82-be27-d149e56f8d6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:52.933369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ef3c5f-0206-4e82-be27-d149e56f8d6f None admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:52.933644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ef3c5f-0206-4e82-be27-d149e56f8d6f None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:52.933911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ef3c5f-0206-4e82-be27-d149e56f8d6f None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c/users/ba2b60a965074d58b885dbc37040042a/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:52.937118 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ef3c5f-0206-4e82-be27-d149e56f8d6f None admin] RBAC: Authorizing `identity:create_grant(project_id=8d4ba5b452824beab548b1646270d12c, user_id=ba2b60a965074d58b885dbc37040042a, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:52.939851 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ef3c5f-0206-4e82-be27-d149e56f8d6f None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:53.015610 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 133/260] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:42:52 2022] PUT /identity/v3/projects/8d4ba5b452824beab548b1646270d12c/users/ba2b60a965074d58b885dbc37040042a/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 13:42:53.022854 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ef3c5f-0206-4e82-be27-d149e56f8d6f None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:53.025172 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 128/261] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:42:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 481 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:42:53.037666 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6038a16f-3ad3-440e-8705-294c77f80160 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:53.040630 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:53.177594 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ef3c5f-0206-4e82-be27-d149e56f8d6f None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:53.196123 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:53.256195 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea4a864-fb1d-4b52-8f11-683f52564705 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:53.257056 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea4a864-fb1d-4b52-8f11-683f52564705 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:53.257337 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea4a864-fb1d-4b52-8f11-683f52564705 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:53.257605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea4a864-fb1d-4b52-8f11-683f52564705 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:53.273233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6038a16f-3ad3-440e-8705-294c77f80160 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:53.286515 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea4a864-fb1d-4b52-8f11-683f52564705 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:53.291464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-8ea4a864-fb1d-4b52-8f11-683f52564705 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:42:53.291464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:53.291464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:53.291464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:53.291464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:53.291464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:53.291464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:53.291464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:53.291464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:53.291464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:53.291464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:53.291464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:53.291464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:53.291464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:53.291464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:53.291464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:53.292572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:53.292572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:53.292572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:53.292572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:53.292572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:53.292572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:53.292572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:53.292572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:53.292572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:53.292572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:53.292572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:53.292572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:53.292572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:53.292572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:53.292572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:53.292572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:53.293372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:53.293372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:53.293372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:53.293372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:53.293372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:53.293372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:53.293372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:53.293372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:53.293372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:53.293372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:53.293372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:42:53.293372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:42:53.293372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 134/262] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:42:53 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:53.303320 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea4a864-fb1d-4b52-8f11-683f52564705 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:53.324046 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d85a2a-4d76-462f-baf8-96c5476a5d7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:53.325061 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d85a2a-4d76-462f-baf8-96c5476a5d7b None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:53.325608 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d85a2a-4d76-462f-baf8-96c5476a5d7b None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:53.326101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d85a2a-4d76-462f-baf8-96c5476a5d7b None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:53.327510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d85a2a-4d76-462f-baf8-96c5476a5d7b None admin] RBAC: Authorizing `identity:get_project(project_id=8d4ba5b452824beab548b1646270d12c)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:53.329489 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d85a2a-4d76-462f-baf8-96c5476a5d7b None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:53.332564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 129/263] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:53 2022] GET /identity/v3/projects/8d4ba5b452824beab548b1646270d12c => generated 302 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:53.342277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d85a2a-4d76-462f-baf8-96c5476a5d7b None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:53.440333 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:53.495720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d85a2a-4d76-462f-baf8-96c5476a5d7b None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:53.522280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea4a864-fb1d-4b52-8f11-683f52564705 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:53.562117 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb674bb9-14d2-4ebf-8771-416a94961ebd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:53.563209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb674bb9-14d2-4ebf-8771-416a94961ebd None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:53.563692 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb674bb9-14d2-4ebf-8771-416a94961ebd None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:53.565235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb674bb9-14d2-4ebf-8771-416a94961ebd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:53.566174 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb674bb9-14d2-4ebf-8771-416a94961ebd None admin] RBAC: Adding query filter params (role.id=727f0fa598e54b9a929d2cf6277c875e, scope.project.id=8d4ba5b452824beab548b1646270d12c, user.id=1ad6dce8df8349ea9c810da3a6b26864) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:53.566719 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb674bb9-14d2-4ebf-8771-416a94961ebd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:53.569042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb674bb9-14d2-4ebf-8771-416a94961ebd None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:53.578198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 130/264] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:42:53 2022] GET /identity/v3/role_assignments?user.id=1ad6dce8df8349ea9c810da3a6b26864&scope.project.id=8d4ba5b452824beab548b1646270d12c&role.id=727f0fa598e54b9a929d2cf6277c875e => generated 608 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:53.582476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273977c0-b1af-4741-bd35-6ad154fdf068 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:53.583507 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273977c0-b1af-4741-bd35-6ad154fdf068 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:53.583995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273977c0-b1af-4741-bd35-6ad154fdf068 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:53.584469 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273977c0-b1af-4741-bd35-6ad154fdf068 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:53.585370 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273977c0-b1af-4741-bd35-6ad154fdf068 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:53.585902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273977c0-b1af-4741-bd35-6ad154fdf068 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:53.587579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273977c0-b1af-4741-bd35-6ad154fdf068 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:53.602642 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 135/265] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:42:53 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:53.609525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb674bb9-14d2-4ebf-8771-416a94961ebd None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:53.747823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb674bb9-14d2-4ebf-8771-416a94961ebd None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:53.791972 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c068eeb-a155-470a-bee5-1a1bb08e4a3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:53.792758 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c068eeb-a155-470a-bee5-1a1bb08e4a3c None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:53.793034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c068eeb-a155-470a-bee5-1a1bb08e4a3c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:53.793295 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c068eeb-a155-470a-bee5-1a1bb08e4a3c None admin] PATH_INFO: `/v3/users/1ad6dce8df8349ea9c810da3a6b26864` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:53.794459 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c068eeb-a155-470a-bee5-1a1bb08e4a3c None admin] RBAC: Authorizing `identity:get_user(user_id=1ad6dce8df8349ea9c810da3a6b26864)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:53.795908 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c068eeb-a155-470a-bee5-1a1bb08e4a3c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:53.797826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 131/266] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:53 2022] GET /identity/v3/users/1ad6dce8df8349ea9c810da3a6b26864 => generated 283 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:53.808033 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273977c0-b1af-4741-bd35-6ad154fdf068 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:53.851046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:53.942977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273977c0-b1af-4741-bd35-6ad154fdf068 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:53.990630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b85c652-9a52-42a2-a901-d8113ad89253 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:53.991245 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b85c652-9a52-42a2-a901-d8113ad89253 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:53.991521 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b85c652-9a52-42a2-a901-d8113ad89253 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:53.991789 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b85c652-9a52-42a2-a901-d8113ad89253 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:54.099204 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b85c652-9a52-42a2-a901-d8113ad89253 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:54.100599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b85c652-9a52-42a2-a901-d8113ad89253 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:54.148175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:54.290955 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 136/267] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:42:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 484 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:42:54.297039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c068eeb-a155-470a-bee5-1a1bb08e4a3c None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:54.304880 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:54.455149 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e19c73-d9fd-499a-a40c-ce4f697aa0de None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:54.455480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e19c73-d9fd-499a-a40c-ce4f697aa0de None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:54.455696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e19c73-d9fd-499a-a40c-ce4f697aa0de None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:54.457689 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 137/268] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:42:54.463005 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c068eeb-a155-470a-bee5-1a1bb08e4a3c None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:54.463967 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5b6f53-6fa8-4494-8951-d597fbe0a669 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:54.464249 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5b6f53-6fa8-4494-8951-d597fbe0a669 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:54.464521 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5b6f53-6fa8-4494-8951-d597fbe0a669 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:54.491221 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-eb5b6f53-6fa8-4494-8951-d597fbe0a669 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:54.492397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-eb5b6f53-6fa8-4494-8951-d597fbe0a669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:54.502405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60638a1c-bba1-437b-9ccd-5a2a8224621e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:54.503355 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60638a1c-bba1-437b-9ccd-5a2a8224621e None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:54.503839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60638a1c-bba1-437b-9ccd-5a2a8224621e None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:54.504307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60638a1c-bba1-437b-9ccd-5a2a8224621e None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:54.505638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60638a1c-bba1-437b-9ccd-5a2a8224621e None admin] RBAC: Authorizing `identity:get_project(project_id=8d4ba5b452824beab548b1646270d12c)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:54.507360 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60638a1c-bba1-437b-9ccd-5a2a8224621e None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:54.509525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 132/269] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:54 2022] GET /identity/v3/projects/8d4ba5b452824beab548b1646270d12c => generated 302 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:54.515982 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf6e495-0c40-4415-ab4f-34b61dac5c5a None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:54.516876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf6e495-0c40-4415-ab4f-34b61dac5c5a None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:54.517354 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf6e495-0c40-4415-ab4f-34b61dac5c5a None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:54.519160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 133/270] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:54 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:54.520097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf6e495-0c40-4415-ab4f-34b61dac5c5a None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:54.578232 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 138/271] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 23 13:42:54.585732 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1407731-c9eb-488b-9e86-6fb6c217d341 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:54.585732 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1407731-c9eb-488b-9e86-6fb6c217d341 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:54.585732 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1407731-c9eb-488b-9e86-6fb6c217d341 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:54.597462 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-d1407731-c9eb-488b-9e86-6fb6c217d341 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:54.598658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d1407731-c9eb-488b-9e86-6fb6c217d341 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:54.631742 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:54.677886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:54.692812 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf6e495-0c40-4415-ab4f-34b61dac5c5a None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:54.702583 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 139/272] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:54.729403 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e23727-cb2e-460f-ae6f-39c9c63bde68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:54.729941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e23727-cb2e-460f-ae6f-39c9c63bde68 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:54.730217 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e23727-cb2e-460f-ae6f-39c9c63bde68 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:54.730511 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e23727-cb2e-460f-ae6f-39c9c63bde68 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:54.731233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e23727-cb2e-460f-ae6f-39c9c63bde68 None admin] RBAC: Adding query filter params (role.id=5ef6f90ed37e44adafc451b081a501d1, scope.project.id=8d4ba5b452824beab548b1646270d12c, user.id=1ad6dce8df8349ea9c810da3a6b26864) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:54.731535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e23727-cb2e-460f-ae6f-39c9c63bde68 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:54.732971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e23727-cb2e-460f-ae6f-39c9c63bde68 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:54.739206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 134/273] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:42:54 2022] GET /identity/v3/role_assignments?user.id=1ad6dce8df8349ea9c810da3a6b26864&scope.project.id=8d4ba5b452824beab548b1646270d12c&role.id=5ef6f90ed37e44adafc451b081a501d1 => generated 608 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:54.981445 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1407731-c9eb-488b-9e86-6fb6c217d341 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:55.015994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e23727-cb2e-460f-ae6f-39c9c63bde68 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:55.109266 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1407731-c9eb-488b-9e86-6fb6c217d341 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:55.143181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e23727-cb2e-460f-ae6f-39c9c63bde68 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:55.158105 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a91ef06f-cc5a-4a74-a7eb-be59e281d07c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:55.158890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91ef06f-cc5a-4a74-a7eb-be59e281d07c None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:55.159169 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91ef06f-cc5a-4a74-a7eb-be59e281d07c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:55.159425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91ef06f-cc5a-4a74-a7eb-be59e281d07c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:55.160517 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91ef06f-cc5a-4a74-a7eb-be59e281d07c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:55.161946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a91ef06f-cc5a-4a74-a7eb-be59e281d07c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:55.163544 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 140/274] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:42:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:55.164945 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bb4ab8-f95f-4a84-9b05-1644091e8e81 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:55.165159 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bb4ab8-f95f-4a84-9b05-1644091e8e81 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:55.165426 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bb4ab8-f95f-4a84-9b05-1644091e8e81 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:55.166660 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 141/275] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:55.169220 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bb4ab8-f95f-4a84-9b05-1644091e8e81 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:55.183805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89b73bf-38b0-419d-b6a0-943a397fd2b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:55.184437 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89b73bf-38b0-419d-b6a0-943a397fd2b8 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:55.184710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89b73bf-38b0-419d-b6a0-943a397fd2b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:55.184981 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89b73bf-38b0-419d-b6a0-943a397fd2b8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:55.186784 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:55.195606 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89b73bf-38b0-419d-b6a0-943a397fd2b8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:55.201712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-d89b73bf-38b0-419d-b6a0-943a397fd2b8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:55.201712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:55.201712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:55.201712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:55.201712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:55.201712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:55.201712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:55.201712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:55.201712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:55.201712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:55.201712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:55.201712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:55.201712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:55.201712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:55.201712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:55.201712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:55.203014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:55.203014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:55.203014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:55.203014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:55.203014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:55.203014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:55.203014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:55.203014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:55.203014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:55.203014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:55.203014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:55.203014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:55.203014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:55.203014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:55.203014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:55.203014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:55.204002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:55.204002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:55.204002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:55.204002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:55.204002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:55.204002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:55.204002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:55.204002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:55.204002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:55.204002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:55.204002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:55.204002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:42:55.204002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 135/276] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:55.204002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86a5edf-9573-412b-a4fc-908a8014f199 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:55.204796 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86a5edf-9573-412b-a4fc-908a8014f199 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:55.204796 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86a5edf-9573-412b-a4fc-908a8014f199 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:55.222912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-a86a5edf-9573-412b-a4fc-908a8014f199 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:55.224200 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a86a5edf-9573-412b-a4fc-908a8014f199 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:55.255120 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:55.300598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bb4ab8-f95f-4a84-9b05-1644091e8e81 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:55.302073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 136/277] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:55.302939 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86a5edf-9573-412b-a4fc-908a8014f199 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:55.350029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad3e069-3c69-4cca-ae5f-68cd1ba5c689 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:55.350663 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad3e069-3c69-4cca-ae5f-68cd1ba5c689 None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:55.350937 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad3e069-3c69-4cca-ae5f-68cd1ba5c689 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:55.351206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad3e069-3c69-4cca-ae5f-68cd1ba5c689 None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:55.351907 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad3e069-3c69-4cca-ae5f-68cd1ba5c689 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:55.353526 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad3e069-3c69-4cca-ae5f-68cd1ba5c689 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:55.419185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86a5edf-9573-412b-a4fc-908a8014f199 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:55.444464 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:55.456209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b756d3-ea9f-4075-82dd-d991d2267e8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:55.456877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b756d3-ea9f-4075-82dd-d991d2267e8f None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:55.457162 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b756d3-ea9f-4075-82dd-d991d2267e8f None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:55.457438 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b756d3-ea9f-4075-82dd-d991d2267e8f None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:55.458178 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b756d3-ea9f-4075-82dd-d991d2267e8f None admin] RBAC: Adding query filter params (name=member) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:55.458577 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b756d3-ea9f-4075-82dd-d991d2267e8f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:55.464415 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b756d3-ea9f-4075-82dd-d991d2267e8f None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:55.465482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 142/278] 10.222.0.62 () {60 vars in 1132 bytes} [Fri Sep 23 13:42:55 2022] POST /identity/v3/users => generated 293 bytes in 296 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 13:42:55.485344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 137/279] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:42:55 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:55.490983 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad3e069-3c69-4cca-ae5f-68cd1ba5c689 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:55.582423 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b756d3-ea9f-4075-82dd-d991d2267e8f None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:55.607203 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad3e069-3c69-4cca-ae5f-68cd1ba5c689 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:55.626754 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:55.655738 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c890d44c-bab8-4fc3-9c59-4ec06b56a411 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:55.656414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c890d44c-bab8-4fc3-9c59-4ec06b56a411 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:55.656696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c890d44c-bab8-4fc3-9c59-4ec06b56a411 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:55.656962 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c890d44c-bab8-4fc3-9c59-4ec06b56a411 None admin] PATH_INFO: `/v3/users/04dfc647d69a414ca594fd886dc3c838` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:55.672411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c890d44c-bab8-4fc3-9c59-4ec06b56a411 None admin] RBAC: Authorizing `identity:get_user(user_id=04dfc647d69a414ca594fd886dc3c838)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:55.674576 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c890d44c-bab8-4fc3-9c59-4ec06b56a411 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:55.676798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 143/280] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:55 2022] GET /identity/v3/users/04dfc647d69a414ca594fd886dc3c838 => generated 293 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:55.683173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c890d44c-bab8-4fc3-9c59-4ec06b56a411 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:55.724460 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b756d3-ea9f-4075-82dd-d991d2267e8f None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:55.768265 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8819893-aff9-4c0b-ba9e-63c17e4793a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:55.768896 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8819893-aff9-4c0b-ba9e-63c17e4793a9 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:55.769173 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8819893-aff9-4c0b-ba9e-63c17e4793a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:55.769436 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8819893-aff9-4c0b-ba9e-63c17e4793a9 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:55.785260 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8819893-aff9-4c0b-ba9e-63c17e4793a9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:55.793366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-e8819893-aff9-4c0b-ba9e-63c17e4793a9 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 23 13:42:55.793366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:55.793366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:55.793366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:55.793366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:55.793366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:55.793366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:55.793366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:55.793366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:55.793366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:55.793366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:55.793366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:55.793366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:55.793366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:55.793366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:55.793366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:55.794236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:55.794236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:55.794236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:55.794236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:55.794236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:55.794236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:55.794236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:55.794236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:55.794236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:55.794236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:55.794236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:55.794236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:55.794236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:55.794236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:55.794236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:55.794236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:55.795371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:55.795371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:55.795371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:55.795371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:55.795371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:55.795371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:55.795371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:55.795371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:55.795371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:55.795371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:55.795371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 23 13:42:55.795371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:42:55.799291 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 138/281] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:55 2022] GET /identity/v3/roles/reader => generated 84 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:42:55.801246 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8819893-aff9-4c0b-ba9e-63c17e4793a9 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:55.811822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c890d44c-bab8-4fc3-9c59-4ec06b56a411 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:55.857099 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccda6a3-ee0a-4838-b3d6-7fdd904317fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:55.858024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccda6a3-ee0a-4838-b3d6-7fdd904317fc None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:55.858306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccda6a3-ee0a-4838-b3d6-7fdd904317fc None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:55.858724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccda6a3-ee0a-4838-b3d6-7fdd904317fc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:55.860531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccda6a3-ee0a-4838-b3d6-7fdd904317fc None admin] RBAC: Adding query filter params (role.id=e8367770d3124f2db1821b6af66bfeab, user.id=04dfc647d69a414ca594fd886dc3c838, scope.system=all) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:55.860531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccda6a3-ee0a-4838-b3d6-7fdd904317fc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:55.862072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccda6a3-ee0a-4838-b3d6-7fdd904317fc None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:55.862571 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:55.869308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 144/282] 10.222.0.62 () {56 vars in 1308 bytes} [Fri Sep 23 13:42:55 2022] GET /identity/v3/role_assignments?user.id=04dfc647d69a414ca594fd886dc3c838&scope.system=all&role.id=e8367770d3124f2db1821b6af66bfeab => generated 226 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:55.940909 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8819893-aff9-4c0b-ba9e-63c17e4793a9 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:55.980263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ae3e76-20e1-45bd-b280-9246bd155239 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:55.980876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ae3e76-20e1-45bd-b280-9246bd155239 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:55.981151 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ae3e76-20e1-45bd-b280-9246bd155239 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:55.981418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ae3e76-20e1-45bd-b280-9246bd155239 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:55.982093 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ae3e76-20e1-45bd-b280-9246bd155239 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:55.982419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ae3e76-20e1-45bd-b280-9246bd155239 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:55.983841 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ae3e76-20e1-45bd-b280-9246bd155239 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:55.993827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 139/283] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:42:55 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:56.000340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccda6a3-ee0a-4838-b3d6-7fdd904317fc None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:56.038877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:56.045038 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5170d591-a5bb-441c-b95c-588694280580 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:56.045321 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5170d591-a5bb-441c-b95c-588694280580 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:56.045580 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5170d591-a5bb-441c-b95c-588694280580 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:56.047229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 140/284] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:42:56.064295 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73213eb0-52c0-4a0b-9b63-e2fbf1646786 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:56.064679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73213eb0-52c0-4a0b-9b63-e2fbf1646786 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:56.064844 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73213eb0-52c0-4a0b-9b63-e2fbf1646786 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:56.092436 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-73213eb0-52c0-4a0b-9b63-e2fbf1646786 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:56.093668 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-73213eb0-52c0-4a0b-9b63-e2fbf1646786 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:56.128056 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccda6a3-ee0a-4838-b3d6-7fdd904317fc None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:56.171086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc25899-a529-46c2-bc29-24dfaa708523 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:56.171733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc25899-a529-46c2-bc29-24dfaa708523 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:56.172008 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc25899-a529-46c2-bc29-24dfaa708523 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:56.172280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc25899-a529-46c2-bc29-24dfaa708523 None admin] PATH_INFO: `/v3/users/0a0dead8d66c4e8395b7a5f3780e7930` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:56.173441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc25899-a529-46c2-bc29-24dfaa708523 None admin] RBAC: Authorizing `identity:get_user(user_id=0a0dead8d66c4e8395b7a5f3780e7930)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:56.174950 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc25899-a529-46c2-bc29-24dfaa708523 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:56.176861 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 145/285] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:55 2022] GET /identity/v3/users/0a0dead8d66c4e8395b7a5f3780e7930 => generated 297 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:56.186313 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 141/286] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 23 13:42:56.187294 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc25899-a529-46c2-bc29-24dfaa708523 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:56.344914 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc25899-a529-46c2-bc29-24dfaa708523 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:56.351359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:56.395767 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6dcfb4-def2-4214-8e2d-c04c923b2806 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:56.396326 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6dcfb4-def2-4214-8e2d-c04c923b2806 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:56.396605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6dcfb4-def2-4214-8e2d-c04c923b2806 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:56.396870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6dcfb4-def2-4214-8e2d-c04c923b2806 None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:56.398015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6dcfb4-def2-4214-8e2d-c04c923b2806 None admin] RBAC: Authorizing `identity:get_project(project_id=8d4ba5b452824beab548b1646270d12c)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:56.399476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f6dcfb4-def2-4214-8e2d-c04c923b2806 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:56.401490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 146/287] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:56 2022] GET /identity/v3/projects/8d4ba5b452824beab548b1646270d12c => generated 302 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:56.419362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73213eb0-52c0-4a0b-9b63-e2fbf1646786 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:56.471591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:56.546608 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73213eb0-52c0-4a0b-9b63-e2fbf1646786 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:56.598071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d73ea5-f496-48a7-b66e-979af6bf342c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:56.599118 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d73ea5-f496-48a7-b66e-979af6bf342c None admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:56.599586 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d73ea5-f496-48a7-b66e-979af6bf342c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:56.600062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d73ea5-f496-48a7-b66e-979af6bf342c None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c/users/0a0dead8d66c4e8395b7a5f3780e7930/roles/2c1b779eac624a728f7e590135c57450` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:56.605830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d73ea5-f496-48a7-b66e-979af6bf342c None admin] RBAC: Authorizing `identity:create_grant(project_id=8d4ba5b452824beab548b1646270d12c, user_id=0a0dead8d66c4e8395b7a5f3780e7930, role_id=2c1b779eac624a728f7e590135c57450)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:56.607595 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d73ea5-f496-48a7-b66e-979af6bf342c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:56.631315 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6dcfb4-def2-4214-8e2d-c04c923b2806 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:56.686031 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 142/288] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:42:56 2022] PUT /identity/v3/projects/8d4ba5b452824beab548b1646270d12c/users/0a0dead8d66c4e8395b7a5f3780e7930/roles/2c1b779eac624a728f7e590135c57450 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:42:56.757881 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f6dcfb4-def2-4214-8e2d-c04c923b2806 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:56.805003 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eeb6e15-b8f9-47b1-a4fb-abdb8e6fbbe9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:56.805662 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eeb6e15-b8f9-47b1-a4fb-abdb8e6fbbe9 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:56.805938 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eeb6e15-b8f9-47b1-a4fb-abdb8e6fbbe9 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:56.806205 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eeb6e15-b8f9-47b1-a4fb-abdb8e6fbbe9 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:56.823308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eeb6e15-b8f9-47b1-a4fb-abdb8e6fbbe9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:56.828459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-8eeb6e15-b8f9-47b1-a4fb-abdb8e6fbbe9 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:56.828459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:56.828459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:56.828459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:56.828459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:56.828459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:56.828459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:56.828459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:56.828459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:56.828459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:56.828459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:56.828459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:56.828459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:56.828459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:56.828459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:56.828459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:56.829379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:56.829379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:56.829379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:56.829379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:56.829379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:56.829379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:56.829379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:56.829379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:56.829379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:56.829379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:56.829379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:56.829379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:56.829379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:56.829379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:56.829379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:56.829379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:56.830164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:56.830164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:56.830164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:56.830164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:56.830164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:56.830164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:56.830164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:56.830164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:56.830164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:56.830164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:56.830164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:56.830164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:42:56.830164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 147/289] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:56 2022] GET /identity/v3/roles/member => generated 84 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:42:56.837576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d73ea5-f496-48a7-b66e-979af6bf342c None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:56.969123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:56.973602 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d73ea5-f496-48a7-b66e-979af6bf342c None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:57.012857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c72fec2-2bd2-4e2f-805b-19ca780a46d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:57.013466 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c72fec2-2bd2-4e2f-805b-19ca780a46d3 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:57.013741 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c72fec2-2bd2-4e2f-805b-19ca780a46d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:57.014005 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c72fec2-2bd2-4e2f-805b-19ca780a46d3 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:57.014700 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c72fec2-2bd2-4e2f-805b-19ca780a46d3 None admin] RBAC: Adding query filter params (name=member) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:57.014997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c72fec2-2bd2-4e2f-805b-19ca780a46d3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:57.016347 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c72fec2-2bd2-4e2f-805b-19ca780a46d3 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:57.029774 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 143/290] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:42:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:57.035722 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eeb6e15-b8f9-47b1-a4fb-abdb8e6fbbe9 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:57.084723 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:57.179435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eeb6e15-b8f9-47b1-a4fb-abdb8e6fbbe9 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:57.231659 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a09b9f-811d-4cb1-a39e-6d3201723cd0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:57.232227 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a09b9f-811d-4cb1-a39e-6d3201723cd0 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:57.232508 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a09b9f-811d-4cb1-a39e-6d3201723cd0 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:57.232782 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a09b9f-811d-4cb1-a39e-6d3201723cd0 None admin] PATH_INFO: `/v3/users/ba2b60a965074d58b885dbc37040042a` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:57.233928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a09b9f-811d-4cb1-a39e-6d3201723cd0 None admin] RBAC: Authorizing `identity:get_user(user_id=ba2b60a965074d58b885dbc37040042a)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:57.235416 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a09b9f-811d-4cb1-a39e-6d3201723cd0 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:57.237000 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 148/291] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:57 2022] GET /identity/v3/users/ba2b60a965074d58b885dbc37040042a => generated 297 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:57.242563 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c72fec2-2bd2-4e2f-805b-19ca780a46d3 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:57.350511 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c72fec2-2bd2-4e2f-805b-19ca780a46d3 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:57.395939 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269a492d-16b4-43a3-89e0-b14cbbefcc6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:57.396616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269a492d-16b4-43a3-89e0-b14cbbefcc6d None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:57.396904 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269a492d-16b4-43a3-89e0-b14cbbefcc6d None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:57.397177 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269a492d-16b4-43a3-89e0-b14cbbefcc6d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:57.437950 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:57.486965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269a492d-16b4-43a3-89e0-b14cbbefcc6d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:57.488354 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269a492d-16b4-43a3-89e0-b14cbbefcc6d None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:57.630995 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 144/292] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:42:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:42:57.637720 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a09b9f-811d-4cb1-a39e-6d3201723cd0 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:57.765234 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:57.773036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a09b9f-811d-4cb1-a39e-6d3201723cd0 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:57.824457 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447faefa-42d3-4aa7-aea9-4792e875ed86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:57.825461 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447faefa-42d3-4aa7-aea9-4792e875ed86 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:57.825942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447faefa-42d3-4aa7-aea9-4792e875ed86 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:57.826486 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447faefa-42d3-4aa7-aea9-4792e875ed86 None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:57.831073 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447faefa-42d3-4aa7-aea9-4792e875ed86 None admin] RBAC: Authorizing `identity:get_project(project_id=8d4ba5b452824beab548b1646270d12c)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:57.832710 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447faefa-42d3-4aa7-aea9-4792e875ed86 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:57.834622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 149/293] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:42:57 2022] GET /identity/v3/projects/8d4ba5b452824beab548b1646270d12c => generated 302 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:57.841678 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269a492d-16b4-43a3-89e0-b14cbbefcc6d None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:57.914844 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:57.970109 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269a492d-16b4-43a3-89e0-b14cbbefcc6d None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:58.016996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d3d229-34d0-4baf-97eb-5da66d141df9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:58.017533 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d3d229-34d0-4baf-97eb-5da66d141df9 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:58.017815 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d3d229-34d0-4baf-97eb-5da66d141df9 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:58.018088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d3d229-34d0-4baf-97eb-5da66d141df9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:58.018844 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d3d229-34d0-4baf-97eb-5da66d141df9 None admin] RBAC: Adding query filter params (role.id=e8367770d3124f2db1821b6af66bfeab, scope.project.id=8d4ba5b452824beab548b1646270d12c, user.id=ba2b60a965074d58b885dbc37040042a) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:58.019150 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d3d229-34d0-4baf-97eb-5da66d141df9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:58.020594 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d3d229-34d0-4baf-97eb-5da66d141df9 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:58.031016 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 145/294] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:42:57 2022] GET /identity/v3/role_assignments?user.id=ba2b60a965074d58b885dbc37040042a&scope.project.id=8d4ba5b452824beab548b1646270d12c&role.id=e8367770d3124f2db1821b6af66bfeab => generated 608 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:58.202319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49df3cb6-6cca-4c0a-b51a-a8fdc4ed6660 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:58.205641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49df3cb6-6cca-4c0a-b51a-a8fdc4ed6660 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:58.205641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49df3cb6-6cca-4c0a-b51a-a8fdc4ed6660 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:58.205641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 150/295] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:58.210588 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97403dde-9d99-4a5e-b051-6ff1c03064b5 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:58.211166 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97403dde-9d99-4a5e-b051-6ff1c03064b5 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:58.211610 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97403dde-9d99-4a5e-b051-6ff1c03064b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:58.254357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-97403dde-9d99-4a5e-b051-6ff1c03064b5 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:58.254357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-97403dde-9d99-4a5e-b051-6ff1c03064b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:58.334904 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 146/296] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:58.545908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b60b5cd-aaee-47ef-ad23-ff044fc22f05 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:58.546262 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b60b5cd-aaee-47ef-ad23-ff044fc22f05 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:58.546492 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b60b5cd-aaee-47ef-ad23-ff044fc22f05 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:58.548068 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 151/297] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:58.553446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cb4fb2-cb25-4aa6-840b-f8dc28a6bcaf None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:58.553914 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cb4fb2-cb25-4aa6-840b-f8dc28a6bcaf None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:58.553990 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cb4fb2-cb25-4aa6-840b-f8dc28a6bcaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:58.574734 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-f1cb4fb2-cb25-4aa6-840b-f8dc28a6bcaf None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:58.575919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f1cb4fb2-cb25-4aa6-840b-f8dc28a6bcaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:58.631764 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:58.645389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b60b5cd-aaee-47ef-ad23-ff044fc22f05 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:58.654883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 147/298] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:42:58.769177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b60b5cd-aaee-47ef-ad23-ff044fc22f05 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:58.811674 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20da84de-ebc6-4652-9287-90f98a4392cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:58.812304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20da84de-ebc6-4652-9287-90f98a4392cd None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:58.812585 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20da84de-ebc6-4652-9287-90f98a4392cd None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:58.812856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20da84de-ebc6-4652-9287-90f98a4392cd None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:58.824740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20da84de-ebc6-4652-9287-90f98a4392cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:58.829726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-20da84de-ebc6-4652-9287-90f98a4392cd None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 23 13:42:58.829726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:58.829726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:58.829726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:58.829726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:58.829726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:58.829726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:58.829726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:58.829726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:58.829726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:58.829726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:58.829726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:58.829726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:58.829726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:58.829726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:58.829726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:58.830652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:58.830652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:58.830652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:58.830652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:58.830652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:58.830652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:58.830652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:58.830652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:58.830652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:58.830652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:58.830652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:58.830652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:58.830652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:58.830652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:58.830652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:58.830652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:58.831439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:58.831439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:58.831439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:58.831439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:58.831439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:58.831439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:58.831439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:58.831439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:58.831439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:58.831439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:58.831439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 23 13:42:58.831439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:42:58.831439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 152/299] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:58 2022] GET /identity/v3/roles/reader => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:42:58.838897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cb4fb2-cb25-4aa6-840b-f8dc28a6bcaf None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:58.899299 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7d0e60-8b78-40c9-8d3b-7d392f64445d None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:58.899606 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7d0e60-8b78-40c9-8d3b-7d392f64445d None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:58.899811 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7d0e60-8b78-40c9-8d3b-7d392f64445d None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:58.901702 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 153/300] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:42:58.907490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbf0fdb-d6ce-49d3-bee7-764ece0b545c None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:58.907892 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbf0fdb-d6ce-49d3-bee7-764ece0b545c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:58.908269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbf0fdb-d6ce-49d3-bee7-764ece0b545c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:58.933309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-8dbf0fdb-d6ce-49d3-bee7-764ece0b545c None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:58.934735 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8dbf0fdb-d6ce-49d3-bee7-764ece0b545c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:58.957431 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cb4fb2-cb25-4aa6-840b-f8dc28a6bcaf None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:58.961392 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:58.997134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a6045c-0467-4939-aade-04cb4fe593ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:58.998360 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a6045c-0467-4939-aade-04cb4fe593ed None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:58.998659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a6045c-0467-4939-aade-04cb4fe593ed None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:58.999084 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a6045c-0467-4939-aade-04cb4fe593ed None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:58.999745 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a6045c-0467-4939-aade-04cb4fe593ed None admin] RBAC: Adding query filter params (name=reader) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:59.000145 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a6045c-0467-4939-aade-04cb4fe593ed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:59.001403 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a6045c-0467-4939-aade-04cb4fe593ed None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:59.009039 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 154/301] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 23 13:42:59.011215 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 148/302] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:42:58 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:59.011964 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbf0fdb-d6ce-49d3-bee7-764ece0b545c None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:59.023311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a6045c-0467-4939-aade-04cb4fe593ed None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:59.077640 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:59.133989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbf0fdb-d6ce-49d3-bee7-764ece0b545c None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:59.147670 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a6045c-0467-4939-aade-04cb4fe593ed None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:59.182951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97265bc2-d66a-43a3-aafb-be751764cc83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:59.183501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97265bc2-d66a-43a3-aafb-be751764cc83 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:59.183775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97265bc2-d66a-43a3-aafb-be751764cc83 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:59.184043 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97265bc2-d66a-43a3-aafb-be751764cc83 None admin] PATH_INFO: `/v3/users/97e59ced50fc4c678b439f7b007315d8` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:59.197468 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97265bc2-d66a-43a3-aafb-be751764cc83 None admin] RBAC: Authorizing `identity:get_user(user_id=97e59ced50fc4c678b439f7b007315d8)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:59.199050 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97265bc2-d66a-43a3-aafb-be751764cc83 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:59.201274 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 149/303] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:59 2022] GET /identity/v3/users/97e59ced50fc4c678b439f7b007315d8 => generated 293 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:59.206789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97265bc2-d66a-43a3-aafb-be751764cc83 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:59.234365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a266ae-f711-4b7a-966e-9587566f3429 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:59.235652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a266ae-f711-4b7a-966e-9587566f3429 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:59.236149 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a266ae-f711-4b7a-966e-9587566f3429 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:59.236755 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a266ae-f711-4b7a-966e-9587566f3429 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:59.238192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a266ae-f711-4b7a-966e-9587566f3429 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:59.240226 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a266ae-f711-4b7a-966e-9587566f3429 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:59.242445 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 155/304] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:42:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:59.248448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a266ae-f711-4b7a-966e-9587566f3429 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:59.279457 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:59.330161 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97265bc2-d66a-43a3-aafb-be751764cc83 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:59.332676 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:59.364147 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a266ae-f711-4b7a-966e-9587566f3429 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:59.367392 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2920a622-bdf0-4ea9-921e-aa54049e2bc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:59.367392 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2920a622-bdf0-4ea9-921e-aa54049e2bc1 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:59.367392 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2920a622-bdf0-4ea9-921e-aa54049e2bc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:59.368493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2920a622-bdf0-4ea9-921e-aa54049e2bc1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:59.369756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2920a622-bdf0-4ea9-921e-aa54049e2bc1 None admin] RBAC: Adding query filter params (role.id=2c1b779eac624a728f7e590135c57450, user.id=97e59ced50fc4c678b439f7b007315d8, scope.system=all) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:59.370064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2920a622-bdf0-4ea9-921e-aa54049e2bc1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:59.371535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2920a622-bdf0-4ea9-921e-aa54049e2bc1 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:59.377007 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 150/305] 10.222.0.62 () {56 vars in 1308 bytes} [Fri Sep 23 13:42:59 2022] GET /identity/v3/role_assignments?user.id=97e59ced50fc4c678b439f7b007315d8&scope.system=all&role.id=2c1b779eac624a728f7e590135c57450 => generated 226 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:42:59.377637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2920a622-bdf0-4ea9-921e-aa54049e2bc1 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:59.423110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adc8603-5314-4a55-b2c4-a46453ea139c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:59.423738 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adc8603-5314-4a55-b2c4-a46453ea139c None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:59.424017 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adc8603-5314-4a55-b2c4-a46453ea139c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:59.424938 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adc8603-5314-4a55-b2c4-a46453ea139c None admin] PATH_INFO: `/v3/groups` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:59.425587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8adc8603-5314-4a55-b2c4-a46453ea139c None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:59.427487 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8adc8603-5314-4a55-b2c4-a46453ea139c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:59.490108 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2920a622-bdf0-4ea9-921e-aa54049e2bc1 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:59.509204 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 156/306] 10.222.0.62 () {60 vars in 1134 bytes} [Fri Sep 23 13:42:59 2022] POST /identity/v3/groups => generated 229 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:42:59.511431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504f1644-5d7b-4fdc-80e2-aefb779e7ad2 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:59.511723 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504f1644-5d7b-4fdc-80e2-aefb779e7ad2 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:59.512025 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504f1644-5d7b-4fdc-80e2-aefb779e7ad2 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:59.518304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 157/307] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:42:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:42:59.524172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8955cd7-1211-4d40-a8c3-c5a36e75a2d6 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:59.524484 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8955cd7-1211-4d40-a8c3-c5a36e75a2d6 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:59.524833 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8955cd7-1211-4d40-a8c3-c5a36e75a2d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:59.550510 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-a8955cd7-1211-4d40-a8c3-c5a36e75a2d6 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:42:59.551751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a8955cd7-1211-4d40-a8c3-c5a36e75a2d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:42:59.552289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8e2296-3614-421c-bfb7-ee1f9f3c23f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:59.553150 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8e2296-3614-421c-bfb7-ee1f9f3c23f0 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:59.553638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8e2296-3614-421c-bfb7-ee1f9f3c23f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:59.554978 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8e2296-3614-421c-bfb7-ee1f9f3c23f0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:59.571659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c8e2296-3614-421c-bfb7-ee1f9f3c23f0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:42:59.578034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-2c8e2296-3614-421c-bfb7-ee1f9f3c23f0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:59.578034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:42:59.578034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:42:59.578034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:42:59.578034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:42:59.578034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:42:59.578034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:42:59.578034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:42:59.578034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:42:59.578034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:42:59.578034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:42:59.578034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:42:59.578034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:42:59.578034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:42:59.578034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:42:59.578034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:42:59.579027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:42:59.579027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:42:59.579027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:42:59.579027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:42:59.579027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:42:59.579027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:42:59.579027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:42:59.579027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:42:59.579027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:42:59.579027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:42:59.579027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:42:59.579027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:42:59.579027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:42:59.579027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:42:59.579027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:42:59.579027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:42:59.579800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:42:59.579800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:42:59.579800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:42:59.579800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:42:59.579800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:42:59.579800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:42:59.579800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:42:59.579800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:42:59.579800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:42:59.579800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:42:59.579800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:42:59.579800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:42:59.582415 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 151/308] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:42:59.591958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8e2296-3614-421c-bfb7-ee1f9f3c23f0 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:59.626025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:59.640520 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 158/309] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:42:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 23 13:42:59.711153 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8e2296-3614-421c-bfb7-ee1f9f3c23f0 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:59.748627 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b5e137-bf5c-41e8-81c7-b6e8e7dee0e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:59.749449 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b5e137-bf5c-41e8-81c7-b6e8e7dee0e6 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:59.749885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b5e137-bf5c-41e8-81c7-b6e8e7dee0e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:59.750296 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b5e137-bf5c-41e8-81c7-b6e8e7dee0e6 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:59.751134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b5e137-bf5c-41e8-81c7-b6e8e7dee0e6 None admin] RBAC: Adding query filter params (name=member) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:42:59.751572 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b5e137-bf5c-41e8-81c7-b6e8e7dee0e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:59.753078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b5e137-bf5c-41e8-81c7-b6e8e7dee0e6 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:59.763085 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 152/310] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:42:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:59.769132 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8955cd7-1211-4d40-a8c3-c5a36e75a2d6 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:59.787815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:42:59.875095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8955cd7-1211-4d40-a8c3-c5a36e75a2d6 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:42:59.910430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be45a591-0d8f-4ddf-a8e9-ee9fff7197c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:42:59.911341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be45a591-0d8f-4ddf-a8e9-ee9fff7197c7 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:42:59.911620 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be45a591-0d8f-4ddf-a8e9-ee9fff7197c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:42:59.911891 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be45a591-0d8f-4ddf-a8e9-ee9fff7197c7 None admin] PATH_INFO: `/v3/users/04dfc647d69a414ca594fd886dc3c838` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:42:59.914552 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be45a591-0d8f-4ddf-a8e9-ee9fff7197c7 None admin] RBAC: Authorizing `identity:get_user(user_id=04dfc647d69a414ca594fd886dc3c838)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:42:59.916723 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be45a591-0d8f-4ddf-a8e9-ee9fff7197c7 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:42:59.918407 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 159/311] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:42:59 2022] GET /identity/v3/users/04dfc647d69a414ca594fd886dc3c838 => generated 293 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:42:59.924169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b5e137-bf5c-41e8-81c7-b6e8e7dee0e6 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:42:59.925380 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be45a591-0d8f-4ddf-a8e9-ee9fff7197c7 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:00.021549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:00.033462 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:00.039124 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be45a591-0d8f-4ddf-a8e9-ee9fff7197c7 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:00.045340 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b5e137-bf5c-41e8-81c7-b6e8e7dee0e6 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:00.081602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b858343e-207c-4dbc-8ea5-d5111d47da15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:00.082319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b858343e-207c-4dbc-8ea5-d5111d47da15 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:00.082595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b858343e-207c-4dbc-8ea5-d5111d47da15 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:00.082865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b858343e-207c-4dbc-8ea5-d5111d47da15 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:00.089609 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df52543-f3fe-4c78-9dc6-48435717e6eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:00.090685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df52543-f3fe-4c78-9dc6-48435717e6eb None admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:00.091178 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df52543-f3fe-4c78-9dc6-48435717e6eb None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:00.091794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6df52543-f3fe-4c78-9dc6-48435717e6eb None admin] PATH_INFO: `/v3/system/users/04dfc647d69a414ca594fd886dc3c838/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:00.093438 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df52543-f3fe-4c78-9dc6-48435717e6eb None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=04dfc647d69a414ca594fd886dc3c838, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:00.095063 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6df52543-f3fe-4c78-9dc6-48435717e6eb None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:00.095594 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b858343e-207c-4dbc-8ea5-d5111d47da15 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:00.101405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-b858343e-207c-4dbc-8ea5-d5111d47da15 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 23 13:43:00.101405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:00.101405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:00.101405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:00.101405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:00.101405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:00.101405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:00.101405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:00.101405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:00.101405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:00.101405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:00.101405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:00.101405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:00.101405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:00.101405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:00.101405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:00.102364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:00.102364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:00.102364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:00.102364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:00.102364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:00.102364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:00.102364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:00.102364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:00.102364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:00.102364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:00.102364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:00.102364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:00.102364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:00.102364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:00.102364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:00.102364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:00.103144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:00.103144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:00.103144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:00.103144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:00.103144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:00.103144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:00.103144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:00.103144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:00.103144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:00.103144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:00.103144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 23 13:43:00.103144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:00.103144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 160/312] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:42:59 2022] GET /identity/v3/roles/reader => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:00.109338 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b858343e-207c-4dbc-8ea5-d5111d47da15 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:00.188725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 153/313] 10.222.0.62 () {58 vars in 1334 bytes} [Fri Sep 23 13:42:59 2022] PUT /identity/v3/system/users/04dfc647d69a414ca594fd886dc3c838/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:43:00.230138 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b858343e-207c-4dbc-8ea5-d5111d47da15 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:00.277546 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b8a040-596f-4e16-80b8-cc001d64418a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:00.278169 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b8a040-596f-4e16-80b8-cc001d64418a None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:00.278419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b8a040-596f-4e16-80b8-cc001d64418a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:00.278676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b8a040-596f-4e16-80b8-cc001d64418a None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:00.279428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b8a040-596f-4e16-80b8-cc001d64418a None admin] RBAC: Adding query filter params (name=reader) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:00.279744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b8a040-596f-4e16-80b8-cc001d64418a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:00.281391 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b8a040-596f-4e16-80b8-cc001d64418a None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:00.292979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 161/314] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:43:00 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:00.298864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df52543-f3fe-4c78-9dc6-48435717e6eb None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:00.415144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6df52543-f3fe-4c78-9dc6-48435717e6eb None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:00.453483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdbfb85-f45c-483f-aca3-d1ae3c8bd431 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:00.454509 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdbfb85-f45c-483f-aca3-d1ae3c8bd431 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:00.454984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdbfb85-f45c-483f-aca3-d1ae3c8bd431 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:00.455509 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdbfb85-f45c-483f-aca3-d1ae3c8bd431 None admin] PATH_INFO: `/v3/users/0a0dead8d66c4e8395b7a5f3780e7930` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:00.456907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdbfb85-f45c-483f-aca3-d1ae3c8bd431 None admin] RBAC: Authorizing `identity:get_user(user_id=0a0dead8d66c4e8395b7a5f3780e7930)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:00.458762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdbfb85-f45c-483f-aca3-d1ae3c8bd431 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:00.465648 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 154/315] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:43:00 2022] GET /identity/v3/users/0a0dead8d66c4e8395b7a5f3780e7930 => generated 297 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:00.467421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b8a040-596f-4e16-80b8-cc001d64418a None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:00.476994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:00.584376 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b8a040-596f-4e16-80b8-cc001d64418a None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:00.622341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd899f49-c269-4548-83e1-b6f964b03e2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:00.623475 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd899f49-c269-4548-83e1-b6f964b03e2e None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:00.623903 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd899f49-c269-4548-83e1-b6f964b03e2e None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:00.624316 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd899f49-c269-4548-83e1-b6f964b03e2e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:00.699868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd899f49-c269-4548-83e1-b6f964b03e2e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:00.701656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd899f49-c269-4548-83e1-b6f964b03e2e None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:00.709994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:00.810859 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 162/316] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:00.816295 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdbfb85-f45c-483f-aca3-d1ae3c8bd431 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:00.846836 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:00.928891 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdbfb85-f45c-483f-aca3-d1ae3c8bd431 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:00.966533 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771a3be9-1895-439b-a437-20304f102f09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:00.967626 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771a3be9-1895-439b-a437-20304f102f09 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:00.968122 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771a3be9-1895-439b-a437-20304f102f09 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:00.968587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771a3be9-1895-439b-a437-20304f102f09 None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:00.969897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771a3be9-1895-439b-a437-20304f102f09 None admin] RBAC: Authorizing `identity:get_project(project_id=8d4ba5b452824beab548b1646270d12c)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:00.971767 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771a3be9-1895-439b-a437-20304f102f09 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:00.973923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 155/317] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:43:00 2022] GET /identity/v3/projects/8d4ba5b452824beab548b1646270d12c => generated 302 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:00.979650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd899f49-c269-4548-83e1-b6f964b03e2e None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:01.082212 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd899f49-c269-4548-83e1-b6f964b03e2e None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:01.090257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:01.117761 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319fa889-ad95-4a89-afb5-739c1e6e2a97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:01.118323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319fa889-ad95-4a89-afb5-739c1e6e2a97 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:01.118622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319fa889-ad95-4a89-afb5-739c1e6e2a97 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:01.118888 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319fa889-ad95-4a89-afb5-739c1e6e2a97 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:01.119759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319fa889-ad95-4a89-afb5-739c1e6e2a97 None admin] RBAC: Adding query filter params (role.id=2c1b779eac624a728f7e590135c57450, scope.project.id=8d4ba5b452824beab548b1646270d12c, user.id=0a0dead8d66c4e8395b7a5f3780e7930) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:01.120063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319fa889-ad95-4a89-afb5-739c1e6e2a97 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:01.121482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319fa889-ad95-4a89-afb5-739c1e6e2a97 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:01.128130 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 163/318] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:43:00 2022] GET /identity/v3/role_assignments?user.id=0a0dead8d66c4e8395b7a5f3780e7930&scope.project.id=8d4ba5b452824beab548b1646270d12c&role.id=2c1b779eac624a728f7e590135c57450 => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:01.670425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556ba86c-c61a-4791-81dc-385e170887b7 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:01.670792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556ba86c-c61a-4791-81dc-385e170887b7 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:01.671091 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556ba86c-c61a-4791-81dc-385e170887b7 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:01.672705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 156/319] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:01.678221 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488bd737-f20e-4298-83dc-1b9dd2a3ad75 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:01.678745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488bd737-f20e-4298-83dc-1b9dd2a3ad75 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:01.678815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488bd737-f20e-4298-83dc-1b9dd2a3ad75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:01.702272 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-488bd737-f20e-4298-83dc-1b9dd2a3ad75 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:01.703494 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-488bd737-f20e-4298-83dc-1b9dd2a3ad75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:01.770904 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 164/320] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:01.793042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6e0b1c-e45c-4f8f-b361-15e2496e080b None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:01.793325 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6e0b1c-e45c-4f8f-b361-15e2496e080b None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:01.793598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6e0b1c-e45c-4f8f-b361-15e2496e080b None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:01.795108 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 157/321] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:01.800021 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182b73ff-a299-4f65-9e14-d4c81ff329ae None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:01.800309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182b73ff-a299-4f65-9e14-d4c81ff329ae None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:01.800580 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182b73ff-a299-4f65-9e14-d4c81ff329ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:01.818530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-182b73ff-a299-4f65-9e14-d4c81ff329ae None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:01.819725 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-182b73ff-a299-4f65-9e14-d4c81ff329ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:01.889525 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 165/322] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:02.042025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6e0b1c-e45c-4f8f-b361-15e2496e080b None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:02.139235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:02.150007 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6e0b1c-e45c-4f8f-b361-15e2496e080b None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:02.163357 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182b73ff-a299-4f65-9e14-d4c81ff329ae None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:02.188513 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e4983d-157b-4647-aa94-cf688447a2e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:02.189140 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4983d-157b-4647-aa94-cf688447a2e5 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:02.189422 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4983d-157b-4647-aa94-cf688447a2e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:02.189690 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e4983d-157b-4647-aa94-cf688447a2e5 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:02.193881 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:02.200887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e4983d-157b-4647-aa94-cf688447a2e5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:02.205785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-79e4983d-157b-4647-aa94-cf688447a2e5 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 23 13:43:02.205785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:02.205785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:02.205785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:02.205785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:02.205785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:02.205785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:02.205785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:02.205785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:02.205785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:02.205785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:02.205785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:02.205785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:02.205785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:02.205785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:02.205785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:02.206898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:02.206898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:02.206898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:02.206898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:02.206898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:02.206898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:02.206898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:02.206898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:02.206898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:02.206898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:02.206898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:02.206898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:02.206898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:02.206898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:02.206898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:02.206898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:02.208051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:02.208051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:02.208051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:02.208051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:02.208051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:02.208051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:02.208051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:02.208051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:02.208051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:02.208051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:02.208051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 23 13:43:02.208051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:02.208051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 158/323] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:02 2022] GET /identity/v3/roles/reader => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:02.215311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e4983d-157b-4647-aa94-cf688447a2e5 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:02.281493 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182b73ff-a299-4f65-9e14-d4c81ff329ae None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:02.317257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419228f9-2557-45fb-aeb2-7e3fc1a4a9a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:02.317808 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419228f9-2557-45fb-aeb2-7e3fc1a4a9a4 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:02.318084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419228f9-2557-45fb-aeb2-7e3fc1a4a9a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:02.318372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419228f9-2557-45fb-aeb2-7e3fc1a4a9a4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:02.319469 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419228f9-2557-45fb-aeb2-7e3fc1a4a9a4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:02.320749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e4983d-157b-4647-aa94-cf688447a2e5 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:02.320916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419228f9-2557-45fb-aeb2-7e3fc1a4a9a4 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:02.326364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 166/324] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:02 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:02.328549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419228f9-2557-45fb-aeb2-7e3fc1a4a9a4 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:02.357806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b21d108-f863-42e6-9294-5571acd12bf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:02.358492 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b21d108-f863-42e6-9294-5571acd12bf1 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:02.358773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b21d108-f863-42e6-9294-5571acd12bf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:02.359043 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b21d108-f863-42e6-9294-5571acd12bf1 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:02.359737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b21d108-f863-42e6-9294-5571acd12bf1 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:02.360037 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b21d108-f863-42e6-9294-5571acd12bf1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:02.361461 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b21d108-f863-42e6-9294-5571acd12bf1 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:02.371469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 159/325] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:43:02 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:02.377431 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b21d108-f863-42e6-9294-5571acd12bf1 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:02.394464 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:02.440568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:02.444919 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419228f9-2557-45fb-aeb2-7e3fc1a4a9a4 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:02.481193 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35032de8-d198-4440-9cba-91d395554404 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:02.481800 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35032de8-d198-4440-9cba-91d395554404 None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:02.482190 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35032de8-d198-4440-9cba-91d395554404 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:02.482496 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35032de8-d198-4440-9cba-91d395554404 None admin] PATH_INFO: `/v3/groups` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:02.483127 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35032de8-d198-4440-9cba-91d395554404 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:02.484620 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35032de8-d198-4440-9cba-91d395554404 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:02.487994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b21d108-f863-42e6-9294-5571acd12bf1 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:02.522506 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5a0565-5c4c-40aa-b57a-ef3e55b6c4c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:02.523135 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5a0565-5c4c-40aa-b57a-ef3e55b6c4c6 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:02.523414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5a0565-5c4c-40aa-b57a-ef3e55b6c4c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:02.523686 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5a0565-5c4c-40aa-b57a-ef3e55b6c4c6 None admin] PATH_INFO: `/v3/users/97e59ced50fc4c678b439f7b007315d8` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:02.524835 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5a0565-5c4c-40aa-b57a-ef3e55b6c4c6 None admin] RBAC: Authorizing `identity:get_user(user_id=97e59ced50fc4c678b439f7b007315d8)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:02.526666 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d5a0565-5c4c-40aa-b57a-ef3e55b6c4c6 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:02.528229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 160/326] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:43:02 2022] GET /identity/v3/users/97e59ced50fc4c678b439f7b007315d8 => generated 293 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:02.529411 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168d8edd-764f-4a1d-b66b-ad87adf6376d None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:02.529691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168d8edd-764f-4a1d-b66b-ad87adf6376d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:02.529958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168d8edd-764f-4a1d-b66b-ad87adf6376d None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:02.531206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 161/327] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:02 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:02.533664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168d8edd-764f-4a1d-b66b-ad87adf6376d None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:02.580739 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 167/328] 10.222.0.62 () {60 vars in 1134 bytes} [Fri Sep 23 13:43:02 2022] POST /identity/v3/groups => generated 226 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:43:02.584093 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f2a409-c171-4de1-a71d-c1bf0bfa5425 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:02.584573 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f2a409-c171-4de1-a71d-c1bf0bfa5425 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:02.585041 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f2a409-c171-4de1-a71d-c1bf0bfa5425 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:02.613335 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-c5f2a409-c171-4de1-a71d-c1bf0bfa5425 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:02.615307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-c5f2a409-c171-4de1-a71d-c1bf0bfa5425 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:02.618601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:02.643529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168d8edd-764f-4a1d-b66b-ad87adf6376d None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:02.682453 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864cda2c-4398-4817-bd2f-75da64a40a61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:02.683121 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864cda2c-4398-4817-bd2f-75da64a40a61 None admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:02.683399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864cda2c-4398-4817-bd2f-75da64a40a61 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:02.683668 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864cda2c-4398-4817-bd2f-75da64a40a61 None admin] PATH_INFO: `/v3/system/users/97e59ced50fc4c678b439f7b007315d8/roles/2c1b779eac624a728f7e590135c57450` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:02.685152 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864cda2c-4398-4817-bd2f-75da64a40a61 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=97e59ced50fc4c678b439f7b007315d8, role_id=2c1b779eac624a728f7e590135c57450)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:02.686582 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864cda2c-4398-4817-bd2f-75da64a40a61 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:02.704484 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 168/329] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:02.749305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 162/330] 10.222.0.62 () {58 vars in 1334 bytes} [Fri Sep 23 13:43:02 2022] PUT /identity/v3/system/users/97e59ced50fc4c678b439f7b007315d8/roles/2c1b779eac624a728f7e590135c57450 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 13:43:02.979678 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f2a409-c171-4de1-a71d-c1bf0bfa5425 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:03.069933 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:03.089529 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f2a409-c171-4de1-a71d-c1bf0bfa5425 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:03.132777 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ba33c7-da77-4638-a8c2-80e007ffe48a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:03.133419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ba33c7-da77-4638-a8c2-80e007ffe48a None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:03.133695 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ba33c7-da77-4638-a8c2-80e007ffe48a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:03.133966 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ba33c7-da77-4638-a8c2-80e007ffe48a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:03.145273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ba33c7-da77-4638-a8c2-80e007ffe48a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:03.152433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-f5ba33c7-da77-4638-a8c2-80e007ffe48a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:43:03.152433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:03.152433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:03.152433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:03.152433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:03.152433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:03.152433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:03.152433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:03.152433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:03.152433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:03.152433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:03.152433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:03.152433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:03.152433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:03.152433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:03.152433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:03.153318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:03.153318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:03.153318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:03.153318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:03.153318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:03.153318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:03.153318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:03.153318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:03.153318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:03.153318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:03.153318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:03.153318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:03.153318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:03.153318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:03.153318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:03.153318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:03.154241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:03.154241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:03.154241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:03.154241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:03.154241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:03.154241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:03.154241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:03.154241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:03.154241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:03.154241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:03.154241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:43:03.154241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:03.154241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 169/331] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:03.159860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864cda2c-4398-4817-bd2f-75da64a40a61 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:03.350295 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864cda2c-4398-4817-bd2f-75da64a40a61 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:03.406071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eec2fba-097d-4b66-9cb4-f0be40555c5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:03.406716 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eec2fba-097d-4b66-9cb4-f0be40555c5c None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:03.406993 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eec2fba-097d-4b66-9cb4-f0be40555c5c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:03.407260 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eec2fba-097d-4b66-9cb4-f0be40555c5c None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:03.407934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eec2fba-097d-4b66-9cb4-f0be40555c5c None admin] RBAC: Adding query filter params (name=member) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:03.408230 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eec2fba-097d-4b66-9cb4-f0be40555c5c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:03.409609 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eec2fba-097d-4b66-9cb4-f0be40555c5c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:03.419265 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 163/332] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:43:03 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:03.424769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ba33c7-da77-4638-a8c2-80e007ffe48a None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:03.608145 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ba33c7-da77-4638-a8c2-80e007ffe48a None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:03.655728 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c88341-f53f-4c98-809a-02feee243401 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:03.656379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c88341-f53f-4c98-809a-02feee243401 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:03.656658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c88341-f53f-4c98-809a-02feee243401 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:03.656931 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c88341-f53f-4c98-809a-02feee243401 None admin] PATH_INFO: `/v3/users/04dfc647d69a414ca594fd886dc3c838` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:03.658106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c88341-f53f-4c98-809a-02feee243401 None admin] RBAC: Authorizing `identity:get_user(user_id=04dfc647d69a414ca594fd886dc3c838)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:03.659590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c88341-f53f-4c98-809a-02feee243401 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:03.661199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 170/333] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:43:03 2022] GET /identity/v3/users/04dfc647d69a414ca594fd886dc3c838 => generated 293 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:03.667695 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eec2fba-097d-4b66-9cb4-f0be40555c5c None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:03.679376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:03.825004 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eec2fba-097d-4b66-9cb4-f0be40555c5c None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:03.873690 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6393ce7-c5cc-4897-ba56-1eb6f9876b65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:03.874284 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6393ce7-c5cc-4897-ba56-1eb6f9876b65 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:03.874587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6393ce7-c5cc-4897-ba56-1eb6f9876b65 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:03.874857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6393ce7-c5cc-4897-ba56-1eb6f9876b65 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:03.875592 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6393ce7-c5cc-4897-ba56-1eb6f9876b65 None admin] RBAC: Adding query filter params (role.id=e8367770d3124f2db1821b6af66bfeab, user.id=04dfc647d69a414ca594fd886dc3c838, scope.system=all) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:03.875893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6393ce7-c5cc-4897-ba56-1eb6f9876b65 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:03.877325 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6393ce7-c5cc-4897-ba56-1eb6f9876b65 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:03.895375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 164/334] 10.222.0.62 () {56 vars in 1308 bytes} [Fri Sep 23 13:43:03 2022] GET /identity/v3/role_assignments?user.id=04dfc647d69a414ca594fd886dc3c838&scope.system=all&role.id=e8367770d3124f2db1821b6af66bfeab => generated 510 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:05.321445 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d42dbb9-e77b-41b2-9684-30e5658f0728 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:05.323415 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d42dbb9-e77b-41b2-9684-30e5658f0728 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:05.323415 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d42dbb9-e77b-41b2-9684-30e5658f0728 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:05.324304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 171/335] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:05.329425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8793badd-6a22-4e69-b459-f4df59caff9d None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:05.329984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8793badd-6a22-4e69-b459-f4df59caff9d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:05.329984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8793badd-6a22-4e69-b459-f4df59caff9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:05.354376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-8793badd-6a22-4e69-b459-f4df59caff9d None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:05.355569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8793badd-6a22-4e69-b459-f4df59caff9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:05.417376 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7d4488-050b-40bb-8d43-eb03bf414bec None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:05.417710 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7d4488-050b-40bb-8d43-eb03bf414bec None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:05.418450 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7d4488-050b-40bb-8d43-eb03bf414bec None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:05.420363 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 172/336] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:05.425831 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ec16cc-5942-47e7-a9f9-38c8c0c3acaa None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:05.426107 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ec16cc-5942-47e7-a9f9-38c8c0c3acaa None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:05.426395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ec16cc-5942-47e7-a9f9-38c8c0c3acaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:05.437860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:05.451242 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 165/337] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:05.463807 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-d7ec16cc-5942-47e7-a9f9-38c8c0c3acaa None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:05.465062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d7ec16cc-5942-47e7-a9f9-38c8c0c3acaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:05.482693 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:05.553497 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 173/338] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:05.868073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8793badd-6a22-4e69-b459-f4df59caff9d None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:05.998715 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8793badd-6a22-4e69-b459-f4df59caff9d None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:06.045383 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bed64a6-a529-47ef-82dd-ffa08f99a65a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:06.046019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bed64a6-a529-47ef-82dd-ffa08f99a65a None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:06.046295 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bed64a6-a529-47ef-82dd-ffa08f99a65a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:06.046592 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bed64a6-a529-47ef-82dd-ffa08f99a65a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:06.063657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bed64a6-a529-47ef-82dd-ffa08f99a65a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:06.068596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-2bed64a6-a529-47ef-82dd-ffa08f99a65a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:43:06.068596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:06.068596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:06.068596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:06.068596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:06.068596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:06.068596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:06.068596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:06.068596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:06.068596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:06.068596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:06.068596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:06.068596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:06.068596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:06.068596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:06.068596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:06.069491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:06.069491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:06.069491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:06.069491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:06.069491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:06.069491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:06.069491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:06.069491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:06.069491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:06.069491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:06.069491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:06.069491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:06.069491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:06.069491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:06.069491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:06.069491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:06.070869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:06.070869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:06.070869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:06.070869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:06.070869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:06.070869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:06.070869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:06.070869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:06.070869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:06.070869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:06.070869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:43:06.070869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:06.070869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 166/339] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:43:05 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:06.075605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ec16cc-5942-47e7-a9f9-38c8c0c3acaa None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:06.088126 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b28a8d9-a98e-4975-9e9e-c87857c30f95 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:06.088423 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b28a8d9-a98e-4975-9e9e-c87857c30f95 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:06.088655 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b28a8d9-a98e-4975-9e9e-c87857c30f95 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:06.090421 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 167/340] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:06.095261 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c635ffe-433e-4137-8658-8c729b4bab4b None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:06.095545 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c635ffe-433e-4137-8658-8c729b4bab4b None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:06.095814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c635ffe-433e-4137-8658-8c729b4bab4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:06.131428 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-8c635ffe-433e-4137-8658-8c729b4bab4b None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:06.132803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8c635ffe-433e-4137-8658-8c729b4bab4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:06.230448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ec16cc-5942-47e7-a9f9-38c8c0c3acaa None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:06.247988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:06.286521 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638e24bc-2a7e-4a79-bb5e-c0db8584191b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:06.287160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638e24bc-2a7e-4a79-bb5e-c0db8584191b None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:06.287438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638e24bc-2a7e-4a79-bb5e-c0db8584191b None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:06.287702 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638e24bc-2a7e-4a79-bb5e-c0db8584191b None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:06.289504 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638e24bc-2a7e-4a79-bb5e-c0db8584191b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:06.289814 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638e24bc-2a7e-4a79-bb5e-c0db8584191b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:06.291941 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638e24bc-2a7e-4a79-bb5e-c0db8584191b None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:06.296483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 168/341] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Sep 23 13:43:06.296483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c635ffe-433e-4137-8658-8c729b4bab4b None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:06.315403 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 174/342] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:06 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:06.340624 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638e24bc-2a7e-4a79-bb5e-c0db8584191b None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:06.354668 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:06.496139 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c635ffe-433e-4137-8658-8c729b4bab4b None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:06.502664 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638e24bc-2a7e-4a79-bb5e-c0db8584191b None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:06.566781 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f2637d-87bf-4ffc-a97b-69d75f728d6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:06.567415 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f2637d-87bf-4ffc-a97b-69d75f728d6f None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:06.567713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f2637d-87bf-4ffc-a97b-69d75f728d6f None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:06.567948 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f2637d-87bf-4ffc-a97b-69d75f728d6f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:06.575022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bb5915-1825-4dbe-9384-ccfa590c91d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:06.575635 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bb5915-1825-4dbe-9384-ccfa590c91d2 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:06.575907 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bb5915-1825-4dbe-9384-ccfa590c91d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:06.576171 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bb5915-1825-4dbe-9384-ccfa590c91d2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:06.594593 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f2637d-87bf-4ffc-a97b-69d75f728d6f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:06.600096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-b5f2637d-87bf-4ffc-a97b-69d75f728d6f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 23 13:43:06.600096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:06.600096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:06.600096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:06.600096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:06.600096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:06.600096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:06.600096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:06.600096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:06.600096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:06.600096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:06.600096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:06.600096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:06.600096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:06.600096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:06.600096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:06.600971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:06.600971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:06.600971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:06.600971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:06.600971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:06.600971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:06.600971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:06.600971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:06.600971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:06.600971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:06.600971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:06.600971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:06.600971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:06.600971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:06.600971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:06.600971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:06.601778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:06.601778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:06.601778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:06.601778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:06.601778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:06.601778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:06.601778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:06.601778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:06.601778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:06.601778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:06.601778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 23 13:43:06.601778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:06.601778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 169/343] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:06 2022] GET /identity/v3/roles/reader => generated 84 bytes in 307 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:06.603473 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e25b3a-a5cc-485d-88ff-f5db30f1a1ec None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:06.603473 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e25b3a-a5cc-485d-88ff-f5db30f1a1ec None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:06.605261 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e25b3a-a5cc-485d-88ff-f5db30f1a1ec None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:06.605261 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 170/344] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:06.611626 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6247e269-8093-475e-a761-c92c824cee37 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:06.611898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6247e269-8093-475e-a761-c92c824cee37 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:06.612160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6247e269-8093-475e-a761-c92c824cee37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:06.642508 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-6247e269-8093-475e-a761-c92c824cee37 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:06.643816 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-6247e269-8093-475e-a761-c92c824cee37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:06.669430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bb5915-1825-4dbe-9384-ccfa590c91d2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:06.670900 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bb5915-1825-4dbe-9384-ccfa590c91d2 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:06.681376 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:06.703965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:06.735182 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 171/345] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:06.736331 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6247e269-8093-475e-a761-c92c824cee37 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:06.826842 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 175/346] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 486 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 23 13:43:06.834280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bb5915-1825-4dbe-9384-ccfa590c91d2 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:06.854623 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6247e269-8093-475e-a761-c92c824cee37 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:06.894052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dad9232-4b9e-4159-b052-58443d638a36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:06.895399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dad9232-4b9e-4159-b052-58443d638a36 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:06.896532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dad9232-4b9e-4159-b052-58443d638a36 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:06.896993 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dad9232-4b9e-4159-b052-58443d638a36 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:06.898517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dad9232-4b9e-4159-b052-58443d638a36 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:06.901893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dad9232-4b9e-4159-b052-58443d638a36 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:06.904108 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dad9232-4b9e-4159-b052-58443d638a36 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:06.918442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 172/347] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:43:06 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:06.924313 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dad9232-4b9e-4159-b052-58443d638a36 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:06.980234 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:07.054595 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dad9232-4b9e-4159-b052-58443d638a36 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:07.066146 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bb5915-1825-4dbe-9384-ccfa590c91d2 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:07.084540 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:07.095939 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ee216a-1741-4c1a-9106-7ebac1449509 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:07.099210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ee216a-1741-4c1a-9106-7ebac1449509 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:07.099210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ee216a-1741-4c1a-9106-7ebac1449509 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:07.099210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ee216a-1741-4c1a-9106-7ebac1449509 None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:07.099210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ee216a-1741-4c1a-9106-7ebac1449509 None admin] RBAC: Authorizing `identity:get_project(project_id=feaf606da67b4ccf9411691df09f4216)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:07.099991 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ee216a-1741-4c1a-9106-7ebac1449509 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:07.102356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 173/348] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:43:06 2022] GET /identity/v3/projects/feaf606da67b4ccf9411691df09f4216 => generated 298 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:07.107512 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ee216a-1741-4c1a-9106-7ebac1449509 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:07.141696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66142501-e97b-47e8-ae54-60b8a40817f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:07.142886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66142501-e97b-47e8-ae54-60b8a40817f5 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:07.143168 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66142501-e97b-47e8-ae54-60b8a40817f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:07.143438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66142501-e97b-47e8-ae54-60b8a40817f5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:07.157776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66142501-e97b-47e8-ae54-60b8a40817f5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:07.165970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-66142501-e97b-47e8-ae54-60b8a40817f5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:43:07.165970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:07.165970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:07.165970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:07.165970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:07.165970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:07.165970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:07.165970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:07.165970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:07.165970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:07.165970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:07.165970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:07.165970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:07.165970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:07.165970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:07.165970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:07.167062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:07.167062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:07.167062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:07.167062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:07.167062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:07.167062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:07.167062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:07.167062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:07.167062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:07.167062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:07.167062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:07.167062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:07.167062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:07.167062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:07.167062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:07.167062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:07.167867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:07.167867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:07.167867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:07.167867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:07.167867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:07.167867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:07.167867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:07.167867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:07.167867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:07.167867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:07.167867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:43:07.167867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:07.167867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 176/349] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:06 2022] GET /identity/v3/roles/member => generated 84 bytes in 340 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:07.169130 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66142501-e97b-47e8-ae54-60b8a40817f5 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:07.257720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ee216a-1741-4c1a-9106-7ebac1449509 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:07.264044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:07.299964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec36b198-e24e-4df4-afb0-7668071699eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:07.300569 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66142501-e97b-47e8-ae54-60b8a40817f5 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:07.301493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec36b198-e24e-4df4-afb0-7668071699eb None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:07.302224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec36b198-e24e-4df4-afb0-7668071699eb None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:07.302776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec36b198-e24e-4df4-afb0-7668071699eb None admin] PATH_INFO: `/v3/users/97e59ced50fc4c678b439f7b007315d8` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:07.304182 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec36b198-e24e-4df4-afb0-7668071699eb None admin] RBAC: Authorizing `identity:get_user(user_id=97e59ced50fc4c678b439f7b007315d8)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:07.305779 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec36b198-e24e-4df4-afb0-7668071699eb None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:07.313868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 174/350] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:43:07 2022] GET /identity/v3/users/97e59ced50fc4c678b439f7b007315d8 => generated 293 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:07.315067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec36b198-e24e-4df4-afb0-7668071699eb None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:07.342215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43bbb4e-90ef-4bbe-9273-0f33e76c3e5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:07.342844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43bbb4e-90ef-4bbe-9273-0f33e76c3e5e None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:07.343154 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43bbb4e-90ef-4bbe-9273-0f33e76c3e5e None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:07.343480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43bbb4e-90ef-4bbe-9273-0f33e76c3e5e None admin] PATH_INFO: `/v3/groups/0ac89e094fe44bf999aa9f30f613fecd` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:07.352397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43bbb4e-90ef-4bbe-9273-0f33e76c3e5e None admin] RBAC: Authorizing `identity:get_group(group_id=0ac89e094fe44bf999aa9f30f613fecd)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:07.354846 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43bbb4e-90ef-4bbe-9273-0f33e76c3e5e None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:07.357435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 177/351] 10.222.0.62 () {56 vars in 1180 bytes} [Fri Sep 23 13:43:07 2022] GET /identity/v3/groups/0ac89e094fe44bf999aa9f30f613fecd => generated 226 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:07.358210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43bbb4e-90ef-4bbe-9273-0f33e76c3e5e None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:07.399239 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:07.455905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec36b198-e24e-4df4-afb0-7668071699eb None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:07.503925 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afac166-5f16-4ab5-a53d-a75c8cde8687 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:07.505332 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43bbb4e-90ef-4bbe-9273-0f33e76c3e5e None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:07.505587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afac166-5f16-4ab5-a53d-a75c8cde8687 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:07.506102 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afac166-5f16-4ab5-a53d-a75c8cde8687 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:07.506600 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afac166-5f16-4ab5-a53d-a75c8cde8687 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:07.519565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9afac166-5f16-4ab5-a53d-a75c8cde8687 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:07.525902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-9afac166-5f16-4ab5-a53d-a75c8cde8687 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:43:07.525902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:07.525902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:07.525902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:07.525902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:07.525902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:07.525902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:07.525902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:07.525902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:07.525902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:07.525902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:07.525902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:07.525902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:07.525902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:07.525902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:07.525902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:07.526897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:07.526897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:07.526897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:07.526897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:07.526897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:07.526897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:07.526897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:07.526897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:07.526897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:07.526897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:07.526897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:07.526897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:07.526897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:07.526897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:07.526897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:07.526897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:07.527672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:07.527672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:07.527672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:07.527672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:07.527672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:07.527672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:07.527672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:07.527672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:07.527672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:07.527672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:07.527672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:43:07.527672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:07.530476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 175/352] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:07 2022] GET /identity/v3/roles/member => generated 84 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:07.531643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afac166-5f16-4ab5-a53d-a75c8cde8687 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:07.552263 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a42667-7cf5-4fca-a031-f397298e458e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:07.552836 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a42667-7cf5-4fca-a031-f397298e458e None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:07.553117 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a42667-7cf5-4fca-a031-f397298e458e None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:07.554387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a42667-7cf5-4fca-a031-f397298e458e None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:07.554387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a42667-7cf5-4fca-a031-f397298e458e None admin] RBAC: Adding query filter params (name=member) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:07.554545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a42667-7cf5-4fca-a031-f397298e458e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:07.556740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a42667-7cf5-4fca-a031-f397298e458e None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:07.567909 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 178/353] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:43:07 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:07.568549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a42667-7cf5-4fca-a031-f397298e458e None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:07.575897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:07.658460 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afac166-5f16-4ab5-a53d-a75c8cde8687 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:07.688998 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:07.699841 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a42667-7cf5-4fca-a031-f397298e458e None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:07.701488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292e7aa8-ee9e-42b8-83a0-ef5cbf76c495 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:07.702012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292e7aa8-ee9e-42b8-83a0-ef5cbf76c495 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:07.702279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292e7aa8-ee9e-42b8-83a0-ef5cbf76c495 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:07.702628 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292e7aa8-ee9e-42b8-83a0-ef5cbf76c495 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:07.703417 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292e7aa8-ee9e-42b8-83a0-ef5cbf76c495 None admin] RBAC: Adding query filter params (role.id=2c1b779eac624a728f7e590135c57450, user.id=97e59ced50fc4c678b439f7b007315d8, scope.system=all) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:07.703722 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292e7aa8-ee9e-42b8-83a0-ef5cbf76c495 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:07.705177 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292e7aa8-ee9e-42b8-83a0-ef5cbf76c495 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:07.721616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 176/354] 10.222.0.62 () {56 vars in 1308 bytes} [Fri Sep 23 13:43:07 2022] GET /identity/v3/role_assignments?user.id=97e59ced50fc4c678b439f7b007315d8&scope.system=all&role.id=2c1b779eac624a728f7e590135c57450 => generated 510 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:07.722647 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292e7aa8-ee9e-42b8-83a0-ef5cbf76c495 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:07.749778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32588321-3313-439b-802d-5c43f3a22850 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:07.751182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32588321-3313-439b-802d-5c43f3a22850 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:07.751539 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32588321-3313-439b-802d-5c43f3a22850 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:07.752035 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32588321-3313-439b-802d-5c43f3a22850 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:07.753038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32588321-3313-439b-802d-5c43f3a22850 None admin] RBAC: Adding query filter params (group.id=0ac89e094fe44bf999aa9f30f613fecd, role.id=5ef6f90ed37e44adafc451b081a501d1, scope.project.id=feaf606da67b4ccf9411691df09f4216) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:07.753350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32588321-3313-439b-802d-5c43f3a22850 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:07.755511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32588321-3313-439b-802d-5c43f3a22850 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:07.764143 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 179/355] 10.222.0.62 () {56 vars in 1376 bytes} [Fri Sep 23 13:43:07 2022] GET /identity/v3/role_assignments?group.id=0ac89e094fe44bf999aa9f30f613fecd&scope.project.id=feaf606da67b4ccf9411691df09f4216&role.id=5ef6f90ed37e44adafc451b081a501d1 => generated 260 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:07.765527 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0dc4e6-a2fa-4c64-b07d-59da49d792d6 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:07.765813 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0dc4e6-a2fa-4c64-b07d-59da49d792d6 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:07.766084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0dc4e6-a2fa-4c64-b07d-59da49d792d6 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:07.770086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 180/356] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:07 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:07.771015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0dc4e6-a2fa-4c64-b07d-59da49d792d6 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:07.864670 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292e7aa8-ee9e-42b8-83a0-ef5cbf76c495 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:07.866663 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:07.906041 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c766b0-6ef6-4743-b303-0a134caf1b67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:07.906701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c766b0-6ef6-4743-b303-0a134caf1b67 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:07.907078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c766b0-6ef6-4743-b303-0a134caf1b67 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:07.907291 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c766b0-6ef6-4743-b303-0a134caf1b67 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:07.908033 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c766b0-6ef6-4743-b303-0a134caf1b67 None admin] RBAC: Adding query filter params (name=member) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:07.908337 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c766b0-6ef6-4743-b303-0a134caf1b67 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:07.909820 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c766b0-6ef6-4743-b303-0a134caf1b67 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:07.919877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 177/357] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:43:07 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:07.920982 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb673d99-b2d9-4655-adac-2c93d0334144 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:07.921266 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb673d99-b2d9-4655-adac-2c93d0334144 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:07.921539 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb673d99-b2d9-4655-adac-2c93d0334144 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:07.924641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0dc4e6-a2fa-4c64-b07d-59da49d792d6 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:07.964567 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-eb673d99-b2d9-4655-adac-2c93d0334144 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:07.965878 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-eb673d99-b2d9-4655-adac-2c93d0334144 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:07.975423 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cbf654-ce9d-4e68-8838-931f1de1a9b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:07.976057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cbf654-ce9d-4e68-8838-931f1de1a9b7 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:07.976332 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cbf654-ce9d-4e68-8838-931f1de1a9b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:07.976600 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cbf654-ce9d-4e68-8838-931f1de1a9b7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:08.013142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:08.077379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41cbf654-ce9d-4e68-8838-931f1de1a9b7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:08.079269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41cbf654-ce9d-4e68-8838-931f1de1a9b7 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:08.098427 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 178/358] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:08.098427 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb673d99-b2d9-4655-adac-2c93d0334144 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:08.204155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:08.221565 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 181/359] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 451 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:08.223738 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3cf752-b21c-4dfa-a50d-24e9368e815e None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:08.226277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3cf752-b21c-4dfa-a50d-24e9368e815e None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:08.226277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3cf752-b21c-4dfa-a50d-24e9368e815e None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:08.226277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 182/360] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:08.227140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3cf752-b21c-4dfa-a50d-24e9368e815e None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:08.232426 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb673d99-b2d9-4655-adac-2c93d0334144 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:08.283062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a584c5-f55a-4d55-9f25-2a1c8aded0d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:08.283683 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a584c5-f55a-4d55-9f25-2a1c8aded0d2 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:08.284007 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a584c5-f55a-4d55-9f25-2a1c8aded0d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:08.284283 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a584c5-f55a-4d55-9f25-2a1c8aded0d2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:08.311818 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:08.367818 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3cf752-b21c-4dfa-a50d-24e9368e815e None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:08.381957 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a584c5-f55a-4d55-9f25-2a1c8aded0d2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:08.384989 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a584c5-f55a-4d55-9f25-2a1c8aded0d2 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:08.410851 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f2d249-fef1-4c4e-9592-b4b2f659518c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:08.411990 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f2d249-fef1-4c4e-9592-b4b2f659518c None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:08.412436 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f2d249-fef1-4c4e-9592-b4b2f659518c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:08.413274 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f2d249-fef1-4c4e-9592-b4b2f659518c None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:08.414837 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f2d249-fef1-4c4e-9592-b4b2f659518c None admin] RBAC: Authorizing `identity:get_project(project_id=feaf606da67b4ccf9411691df09f4216)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:08.416731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f2d249-fef1-4c4e-9592-b4b2f659518c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:08.418816 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 183/361] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:43:08 2022] GET /identity/v3/projects/feaf606da67b4ccf9411691df09f4216 => generated 298 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:08.420801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3d66a9-4f33-4ff1-ad81-de2d5df71b80 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:08.421224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3d66a9-4f33-4ff1-ad81-de2d5df71b80 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:08.421630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd3d66a9-4f33-4ff1-ad81-de2d5df71b80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:08.444011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-fd3d66a9-4f33-4ff1-ad81-de2d5df71b80 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:08.445550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-fd3d66a9-4f33-4ff1-ad81-de2d5df71b80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:08.481122 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:08.504782 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 179/362] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:08.506689 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a584c5-f55a-4d55-9f25-2a1c8aded0d2 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:08.542235 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 184/363] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:08.543317 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3d66a9-4f33-4ff1-ad81-de2d5df71b80 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:08.607971 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:08.647878 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a584c5-f55a-4d55-9f25-2a1c8aded0d2 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:08.660903 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd3d66a9-4f33-4ff1-ad81-de2d5df71b80 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:08.692986 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb7af5f-9c40-4262-989b-5dc4b5b59b9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:08.696141 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb7af5f-9c40-4262-989b-5dc4b5b59b9b None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:08.697234 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb7af5f-9c40-4262-989b-5dc4b5b59b9b None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:08.697844 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb7af5f-9c40-4262-989b-5dc4b5b59b9b None admin] PATH_INFO: `/v3/groups/0ac89e094fe44bf999aa9f30f613fecd` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:08.699354 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb7af5f-9c40-4262-989b-5dc4b5b59b9b None admin] RBAC: Authorizing `identity:get_group(group_id=0ac89e094fe44bf999aa9f30f613fecd)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:08.701413 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb7af5f-9c40-4262-989b-5dc4b5b59b9b None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:08.702827 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766f6d07-368e-4f5e-aad7-5a9c9d791f22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:08.703471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766f6d07-368e-4f5e-aad7-5a9c9d791f22 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:08.703798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766f6d07-368e-4f5e-aad7-5a9c9d791f22 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:08.704155 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766f6d07-368e-4f5e-aad7-5a9c9d791f22 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:08.704569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 180/364] 10.222.0.62 () {56 vars in 1180 bytes} [Fri Sep 23 13:43:08 2022] GET /identity/v3/groups/0ac89e094fe44bf999aa9f30f613fecd => generated 226 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:08.705712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb7af5f-9c40-4262-989b-5dc4b5b59b9b None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:08.717504 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766f6d07-368e-4f5e-aad7-5a9c9d791f22 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:08.723398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-766f6d07-368e-4f5e-aad7-5a9c9d791f22 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:43:08.723398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:08.723398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:08.723398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:08.723398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:08.723398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:08.723398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:08.723398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:08.723398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:08.723398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:08.723398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:08.723398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:08.723398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:08.723398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:08.723398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:08.723398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:08.724257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:08.724257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:08.724257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:08.724257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:08.724257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:08.724257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:08.724257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:08.724257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:08.724257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:08.724257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:08.724257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:08.724257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:08.724257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:08.724257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:08.724257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:08.724257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:08.725057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:08.725057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:08.725057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:08.725057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:08.725057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:08.725057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:08.725057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:08.725057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:08.725057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:08.725057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:08.725057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:43:08.725057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:08.725057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 185/365] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:43:08 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:08.725708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766f6d07-368e-4f5e-aad7-5a9c9d791f22 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:08.770716 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:08.829358 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb7af5f-9c40-4262-989b-5dc4b5b59b9b None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:08.845658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766f6d07-368e-4f5e-aad7-5a9c9d791f22 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:08.866218 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:08.874357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8498839-a9a8-4576-81c6-d8becea1deb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:08.875987 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8498839-a9a8-4576-81c6-d8becea1deb9 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:08.875987 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8498839-a9a8-4576-81c6-d8becea1deb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:08.875987 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8498839-a9a8-4576-81c6-d8becea1deb9 None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:08.877310 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8498839-a9a8-4576-81c6-d8becea1deb9 None admin] RBAC: Authorizing `identity:get_project(project_id=8d4ba5b452824beab548b1646270d12c)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:08.879531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8498839-a9a8-4576-81c6-d8becea1deb9 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:08.880728 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 181/366] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:43:08 2022] GET /identity/v3/projects/8d4ba5b452824beab548b1646270d12c => generated 302 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:08.881363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8498839-a9a8-4576-81c6-d8becea1deb9 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:08.903412 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8f7683-db7f-440f-aa71-400f5fe09c24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:08.904330 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8f7683-db7f-440f-aa71-400f5fe09c24 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:08.904671 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8f7683-db7f-440f-aa71-400f5fe09c24 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:08.904871 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8f7683-db7f-440f-aa71-400f5fe09c24 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:08.905596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8f7683-db7f-440f-aa71-400f5fe09c24 None admin] RBAC: Adding query filter params (group.id=0ac89e094fe44bf999aa9f30f613fecd, role.id=e8367770d3124f2db1821b6af66bfeab, scope.project.id=feaf606da67b4ccf9411691df09f4216) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:08.905896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8f7683-db7f-440f-aa71-400f5fe09c24 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:08.907358 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8f7683-db7f-440f-aa71-400f5fe09c24 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:08.918612 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 186/367] 10.222.0.62 () {56 vars in 1376 bytes} [Fri Sep 23 13:43:08 2022] GET /identity/v3/role_assignments?group.id=0ac89e094fe44bf999aa9f30f613fecd&scope.project.id=feaf606da67b4ccf9411691df09f4216&role.id=e8367770d3124f2db1821b6af66bfeab => generated 260 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:08.919722 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8f7683-db7f-440f-aa71-400f5fe09c24 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:09.004168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8498839-a9a8-4576-81c6-d8becea1deb9 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:09.049666 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:09.061091 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b66272-fee4-426e-b6f7-b211e46aebd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:09.061761 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b66272-fee4-426e-b6f7-b211e46aebd1 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:09.061997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b66272-fee4-426e-b6f7-b211e46aebd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:09.063262 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b66272-fee4-426e-b6f7-b211e46aebd1 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:09.063262 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b66272-fee4-426e-b6f7-b211e46aebd1 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:09.063262 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b66272-fee4-426e-b6f7-b211e46aebd1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:09.064627 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b66272-fee4-426e-b6f7-b211e46aebd1 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:09.066973 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8f7683-db7f-440f-aa71-400f5fe09c24 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:09.081397 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 182/368] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:08 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:09.081397 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b66272-fee4-426e-b6f7-b211e46aebd1 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:09.117071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5484566-deec-4630-9676-eb8c040115ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:09.118084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5484566-deec-4630-9676-eb8c040115ef None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:09.119178 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5484566-deec-4630-9676-eb8c040115ef None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:09.119671 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5484566-deec-4630-9676-eb8c040115ef None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:09.137545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5484566-deec-4630-9676-eb8c040115ef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:09.141217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-c5484566-deec-4630-9676-eb8c040115ef None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:43:09.141217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:09.141217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:09.141217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:09.141217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:09.141217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:09.141217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:09.141217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:09.141217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:09.141217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:09.141217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:09.141217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:09.141217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:09.141217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:09.141217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:09.141217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:09.142095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:09.142095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:09.142095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:09.142095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:09.142095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:09.142095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:09.142095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:09.142095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:09.142095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:09.142095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:09.142095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:09.142095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:09.142095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:09.142095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:09.142095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:09.142095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:09.142920 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:09.142920 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:09.142920 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:09.142920 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:09.142920 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:09.142920 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:09.142920 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:09.142920 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:09.142920 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:09.142920 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:09.142920 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:43:09.142920 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:09.143503 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 187/369] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:43:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:09.144447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5484566-deec-4630-9676-eb8c040115ef None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:09.202408 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:09.207524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b66272-fee4-426e-b6f7-b211e46aebd1 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:09.251445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c0b499-5114-4958-99e6-2f18e86489cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:09.252132 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c0b499-5114-4958-99e6-2f18e86489cd None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:09.252413 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c0b499-5114-4958-99e6-2f18e86489cd None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:09.252681 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c0b499-5114-4958-99e6-2f18e86489cd None admin] PATH_INFO: `/v3/groups/0ac89e094fe44bf999aa9f30f613fecd` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:09.254283 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c0b499-5114-4958-99e6-2f18e86489cd None admin] RBAC: Authorizing `identity:get_group(group_id=0ac89e094fe44bf999aa9f30f613fecd)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:09.256137 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c0b499-5114-4958-99e6-2f18e86489cd None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:09.258267 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 183/370] 10.222.0.62 () {56 vars in 1180 bytes} [Fri Sep 23 13:43:09 2022] GET /identity/v3/groups/0ac89e094fe44bf999aa9f30f613fecd => generated 226 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:09.263316 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c0b499-5114-4958-99e6-2f18e86489cd None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:09.291245 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5484566-deec-4630-9676-eb8c040115ef None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:09.339741 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:09.348679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b4f793-9fcc-4b3b-a040-d9d55ae896a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:09.350089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b4f793-9fcc-4b3b-a040-d9d55ae896a0 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:09.351779 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b4f793-9fcc-4b3b-a040-d9d55ae896a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:09.351779 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b4f793-9fcc-4b3b-a040-d9d55ae896a0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:09.415042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c0b499-5114-4958-99e6-2f18e86489cd None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:09.449867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b4f793-9fcc-4b3b-a040-d9d55ae896a0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:09.451468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b4f793-9fcc-4b3b-a040-d9d55ae896a0 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:09.470007 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd58263a-b7c8-4b78-b228-0ca9d4b931ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:09.470654 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd58263a-b7c8-4b78-b228-0ca9d4b931ea None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:09.470935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd58263a-b7c8-4b78-b228-0ca9d4b931ea None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:09.471199 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd58263a-b7c8-4b78-b228-0ca9d4b931ea None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:09.472193 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd58263a-b7c8-4b78-b228-0ca9d4b931ea None admin] RBAC: Adding query filter params (name=admin) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:09.472193 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd58263a-b7c8-4b78-b228-0ca9d4b931ea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:09.473642 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd58263a-b7c8-4b78-b228-0ca9d4b931ea None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:09.485524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 184/371] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:09.486681 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd58263a-b7c8-4b78-b228-0ca9d4b931ea None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:09.502309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:09.567378 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 188/372] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:09.568435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b4f793-9fcc-4b3b-a040-d9d55ae896a0 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:09.639996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd58263a-b7c8-4b78-b228-0ca9d4b931ea None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:09.655373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:09.680673 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819ddafe-8b18-42ca-82c0-65e138ae44c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:09.681566 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819ddafe-8b18-42ca-82c0-65e138ae44c7 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:09.681839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819ddafe-8b18-42ca-82c0-65e138ae44c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:09.682102 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819ddafe-8b18-42ca-82c0-65e138ae44c7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:09.683372 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819ddafe-8b18-42ca-82c0-65e138ae44c7 None admin] RBAC: Adding query filter params (group.id=0ac89e094fe44bf999aa9f30f613fecd, role.id=e8367770d3124f2db1821b6af66bfeab, scope.project.id=8d4ba5b452824beab548b1646270d12c) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:09.683810 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819ddafe-8b18-42ca-82c0-65e138ae44c7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:09.685551 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819ddafe-8b18-42ca-82c0-65e138ae44c7 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:09.692243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 185/373] 10.222.0.62 () {56 vars in 1376 bytes} [Fri Sep 23 13:43:09 2022] GET /identity/v3/role_assignments?group.id=0ac89e094fe44bf999aa9f30f613fecd&scope.project.id=8d4ba5b452824beab548b1646270d12c&role.id=e8367770d3124f2db1821b6af66bfeab => generated 260 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:09.693050 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819ddafe-8b18-42ca-82c0-65e138ae44c7 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:09.708969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b4f793-9fcc-4b3b-a040-d9d55ae896a0 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:09.770948 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c823c38b-0a43-4f13-94f3-ee04b5206258 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:09.771567 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c823c38b-0a43-4f13-94f3-ee04b5206258 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:09.771846 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c823c38b-0a43-4f13-94f3-ee04b5206258 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:09.772596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c823c38b-0a43-4f13-94f3-ee04b5206258 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:09.787866 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:09.832128 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819ddafe-8b18-42ca-82c0-65e138ae44c7 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:09.873377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee89affa-b86f-41d2-8370-64631232cff6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:09.874013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee89affa-b86f-41d2-8370-64631232cff6 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:09.874290 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee89affa-b86f-41d2-8370-64631232cff6 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:09.874587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee89affa-b86f-41d2-8370-64631232cff6 None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:09.875758 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee89affa-b86f-41d2-8370-64631232cff6 None admin] RBAC: Authorizing `identity:get_project(project_id=8d4ba5b452824beab548b1646270d12c)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:09.877196 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee89affa-b86f-41d2-8370-64631232cff6 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:09.877543 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c823c38b-0a43-4f13-94f3-ee04b5206258 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:09.879306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c823c38b-0a43-4f13-94f3-ee04b5206258 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:09.880200 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 186/374] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:43:09 2022] GET /identity/v3/projects/8d4ba5b452824beab548b1646270d12c => generated 302 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:09.886119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee89affa-b86f-41d2-8370-64631232cff6 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:09.944301 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:10.032954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 189/375] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 465 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:10.034373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee89affa-b86f-41d2-8370-64631232cff6 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:10.038147 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c823c38b-0a43-4f13-94f3-ee04b5206258 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:10.080744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcc4903-83eb-479b-9211-9cd327c20e17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:10.081701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcc4903-83eb-479b-9211-9cd327c20e17 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:10.081979 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcc4903-83eb-479b-9211-9cd327c20e17 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:10.082313 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcc4903-83eb-479b-9211-9cd327c20e17 None admin] PATH_INFO: `/v3/groups/0ac89e094fe44bf999aa9f30f613fecd` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:10.083632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcc4903-83eb-479b-9211-9cd327c20e17 None admin] RBAC: Authorizing `identity:get_group(group_id=0ac89e094fe44bf999aa9f30f613fecd)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:10.085662 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcc4903-83eb-479b-9211-9cd327c20e17 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:10.087758 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 187/376] 10.222.0.62 () {56 vars in 1180 bytes} [Fri Sep 23 13:43:09 2022] GET /identity/v3/groups/0ac89e094fe44bf999aa9f30f613fecd => generated 226 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:10.093437 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcc4903-83eb-479b-9211-9cd327c20e17 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:10.101933 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:10.175770 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c823c38b-0a43-4f13-94f3-ee04b5206258 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:10.214713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcc4903-83eb-479b-9211-9cd327c20e17 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:10.222375 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd91455-e347-4867-96bf-16f017c40a3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:10.222375 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd91455-e347-4867-96bf-16f017c40a3f None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:10.222375 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd91455-e347-4867-96bf-16f017c40a3f None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:10.222375 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd91455-e347-4867-96bf-16f017c40a3f None admin] PATH_INFO: `/v3/projects/e15d4d7f53c543e88cdf9075fbadb3c8` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:10.222714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd91455-e347-4867-96bf-16f017c40a3f None admin] RBAC: Authorizing `identity:get_project(project_id=e15d4d7f53c543e88cdf9075fbadb3c8)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:10.222945 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd91455-e347-4867-96bf-16f017c40a3f None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:10.225088 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 190/377] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:43:10 2022] GET /identity/v3/projects/e15d4d7f53c543e88cdf9075fbadb3c8 => generated 344 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:10.226018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:10.232144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd91455-e347-4867-96bf-16f017c40a3f None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:10.254108 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1beed889-fd7f-4eba-8bd7-e28e959a3650 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:10.254758 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1beed889-fd7f-4eba-8bd7-e28e959a3650 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:10.255133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1beed889-fd7f-4eba-8bd7-e28e959a3650 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:10.255773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1beed889-fd7f-4eba-8bd7-e28e959a3650 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:10.256699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1beed889-fd7f-4eba-8bd7-e28e959a3650 None admin] RBAC: Adding query filter params (group.id=0ac89e094fe44bf999aa9f30f613fecd, role.id=5ef6f90ed37e44adafc451b081a501d1, scope.project.id=8d4ba5b452824beab548b1646270d12c) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:10.257206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1beed889-fd7f-4eba-8bd7-e28e959a3650 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:10.259507 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1beed889-fd7f-4eba-8bd7-e28e959a3650 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:10.265790 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 188/378] 10.222.0.62 () {56 vars in 1376 bytes} [Fri Sep 23 13:43:10 2022] GET /identity/v3/role_assignments?group.id=0ac89e094fe44bf999aa9f30f613fecd&scope.project.id=8d4ba5b452824beab548b1646270d12c&role.id=5ef6f90ed37e44adafc451b081a501d1 => generated 260 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:10.378972 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd91455-e347-4867-96bf-16f017c40a3f None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:10.389971 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:10.418822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e056f9-4b9f-4b45-9022-cd8d2beadc43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:10.419488 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e056f9-4b9f-4b45-9022-cd8d2beadc43 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:10.419764 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e056f9-4b9f-4b45-9022-cd8d2beadc43 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:10.420040 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e056f9-4b9f-4b45-9022-cd8d2beadc43 None admin] PATH_INFO: `/v3/groups/023e6ecb4fb747e5ae691eb4cf72e500` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:10.426277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e056f9-4b9f-4b45-9022-cd8d2beadc43 None admin] RBAC: Authorizing `identity:get_group(group_id=023e6ecb4fb747e5ae691eb4cf72e500)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:10.427801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e056f9-4b9f-4b45-9022-cd8d2beadc43 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:10.429834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 191/379] 10.222.0.62 () {56 vars in 1180 bytes} [Fri Sep 23 13:43:10 2022] GET /identity/v3/groups/023e6ecb4fb747e5ae691eb4cf72e500 => generated 229 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:10.436502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1beed889-fd7f-4eba-8bd7-e28e959a3650 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:10.561777 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c88eb0-8722-47ec-9b83-c2a17fa24a0a None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:10.562261 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c88eb0-8722-47ec-9b83-c2a17fa24a0a None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:10.562458 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c88eb0-8722-47ec-9b83-c2a17fa24a0a None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:10.564279 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 192/380] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:43:10.570215 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1beed889-fd7f-4eba-8bd7-e28e959a3650 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:10.587430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca8341c-83a5-4a12-bd86-eaf2961fc9ad None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:10.587691 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca8341c-83a5-4a12-bd86-eaf2961fc9ad None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:10.587963 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca8341c-83a5-4a12-bd86-eaf2961fc9ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:10.614531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-6ca8341c-83a5-4a12-bd86-eaf2961fc9ad None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:10.616148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-6ca8341c-83a5-4a12-bd86-eaf2961fc9ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:10.622028 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77fc4a7-6824-41b4-8253-905979792869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:10.622937 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77fc4a7-6824-41b4-8253-905979792869 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:10.623212 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77fc4a7-6824-41b4-8253-905979792869 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:10.623478 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77fc4a7-6824-41b4-8253-905979792869 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:10.624209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77fc4a7-6824-41b4-8253-905979792869 None admin] RBAC: Adding query filter params (group.id=023e6ecb4fb747e5ae691eb4cf72e500, role.id=727f0fa598e54b9a929d2cf6277c875e, scope.project.id=e15d4d7f53c543e88cdf9075fbadb3c8) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:10.624513 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77fc4a7-6824-41b4-8253-905979792869 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:10.626039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77fc4a7-6824-41b4-8253-905979792869 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:10.633126 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 189/381] 10.222.0.62 () {56 vars in 1376 bytes} [Fri Sep 23 13:43:10 2022] GET /identity/v3/role_assignments?group.id=023e6ecb4fb747e5ae691eb4cf72e500&scope.project.id=e15d4d7f53c543e88cdf9075fbadb3c8&role.id=727f0fa598e54b9a929d2cf6277c875e => generated 260 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:10.716683 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 193/382] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:11.166343 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77fc4a7-6824-41b4-8253-905979792869 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:11.220413 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:11.285881 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77fc4a7-6824-41b4-8253-905979792869 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:11.323423 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b7a163-2e8d-495e-bb80-790193d8718a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:11.324058 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b7a163-2e8d-495e-bb80-790193d8718a None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:11.324334 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b7a163-2e8d-495e-bb80-790193d8718a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:11.324600 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b7a163-2e8d-495e-bb80-790193d8718a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:11.335486 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b7a163-2e8d-495e-bb80-790193d8718a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:11.341184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-50b7a163-2e8d-495e-bb80-790193d8718a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:43:11.341184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:11.341184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:11.341184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:11.341184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:11.341184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:11.341184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:11.341184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:11.341184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:11.341184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:11.341184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:11.341184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:11.341184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:11.341184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:11.341184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:11.341184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:11.342042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:11.342042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:11.342042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:11.342042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:11.342042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:11.342042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:11.342042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:11.342042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:11.342042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:11.342042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:11.342042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:11.342042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:11.342042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:11.342042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:11.342042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:11.342042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:11.342882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:11.342882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:11.342882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:11.342882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:11.342882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:11.342882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:11.342882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:11.342882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:11.342882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:11.342882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:11.342882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:43:11.342882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:11.342882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 190/383] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:43:11 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:11.348146 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca8341c-83a5-4a12-bd86-eaf2961fc9ad None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:11.466166 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca8341c-83a5-4a12-bd86-eaf2961fc9ad None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:11.467819 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:11.503901 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c4bb93-4b8c-43a6-9f90-2b4bbc89abc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:11.504511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c4bb93-4b8c-43a6-9f90-2b4bbc89abc2 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:11.504794 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c4bb93-4b8c-43a6-9f90-2b4bbc89abc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:11.505063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c4bb93-4b8c-43a6-9f90-2b4bbc89abc2 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:11.505725 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c4bb93-4b8c-43a6-9f90-2b4bbc89abc2 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:11.506023 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c4bb93-4b8c-43a6-9f90-2b4bbc89abc2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:11.507452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c4bb93-4b8c-43a6-9f90-2b4bbc89abc2 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:11.516819 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 194/384] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:11 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:11.522351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b7a163-2e8d-495e-bb80-790193d8718a None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:11.637647 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:11.643587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b7a163-2e8d-495e-bb80-790193d8718a None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:11.688704 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e75fc2-95a1-4646-8b42-c836eedd1531 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:11.689218 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e75fc2-95a1-4646-8b42-c836eedd1531 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:11.689218 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e75fc2-95a1-4646-8b42-c836eedd1531 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:11.690601 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 195/385] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:43:11.696546 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bee4ea-eb64-4cc1-816e-7a3504e099c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:11.696936 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3c5cff-fee3-4f9d-a9d3-b85f5d031303 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:11.697410 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3c5cff-fee3-4f9d-a9d3-b85f5d031303 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:11.697530 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bee4ea-eb64-4cc1-816e-7a3504e099c1 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:11.697687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bee4ea-eb64-4cc1-816e-7a3504e099c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:11.697963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bee4ea-eb64-4cc1-816e-7a3504e099c1 None admin] PATH_INFO: `/v3/groups/0ac89e094fe44bf999aa9f30f613fecd` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:11.698345 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3c5cff-fee3-4f9d-a9d3-b85f5d031303 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:11.699164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bee4ea-eb64-4cc1-816e-7a3504e099c1 None admin] RBAC: Authorizing `identity:get_group(group_id=0ac89e094fe44bf999aa9f30f613fecd)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:11.700592 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bee4ea-eb64-4cc1-816e-7a3504e099c1 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:11.706350 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 191/386] 10.222.0.62 () {56 vars in 1180 bytes} [Fri Sep 23 13:43:11 2022] GET /identity/v3/groups/0ac89e094fe44bf999aa9f30f613fecd => generated 226 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:11.707806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bee4ea-eb64-4cc1-816e-7a3504e099c1 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:11.733137 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-cc3c5cff-fee3-4f9d-a9d3-b85f5d031303 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:11.735772 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-cc3c5cff-fee3-4f9d-a9d3-b85f5d031303 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:11.868890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 196/387] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:11.871122 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bee4ea-eb64-4cc1-816e-7a3504e099c1 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:11.924713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc8010d-6c73-468c-b945-b0a2c6509a66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:11.925718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc8010d-6c73-468c-b945-b0a2c6509a66 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:11.926205 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc8010d-6c73-468c-b945-b0a2c6509a66 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:11.926725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc8010d-6c73-468c-b945-b0a2c6509a66 None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:11.928099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc8010d-6c73-468c-b945-b0a2c6509a66 None admin] RBAC: Authorizing `identity:get_project(project_id=feaf606da67b4ccf9411691df09f4216)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:11.929779 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc8010d-6c73-468c-b945-b0a2c6509a66 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:11.933100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 192/388] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:43:11 2022] GET /identity/v3/projects/feaf606da67b4ccf9411691df09f4216 => generated 298 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:11.939768 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3c5cff-fee3-4f9d-a9d3-b85f5d031303 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:12.065125 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3c5cff-fee3-4f9d-a9d3-b85f5d031303 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:12.066708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:12.109906 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9542c66-8b16-4c37-9cfd-547953c135fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:12.110601 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9542c66-8b16-4c37-9cfd-547953c135fe None admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:12.110877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9542c66-8b16-4c37-9cfd-547953c135fe None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:12.111147 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9542c66-8b16-4c37-9cfd-547953c135fe None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216/groups/0ac89e094fe44bf999aa9f30f613fecd/roles/5ef6f90ed37e44adafc451b081a501d1` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:12.113321 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9542c66-8b16-4c37-9cfd-547953c135fe None admin] RBAC: Authorizing `identity:create_grant(project_id=feaf606da67b4ccf9411691df09f4216, group_id=0ac89e094fe44bf999aa9f30f613fecd, role_id=5ef6f90ed37e44adafc451b081a501d1)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:12.114781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9542c66-8b16-4c37-9cfd-547953c135fe None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:12.182894 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 197/389] 10.222.0.62 () {58 vars in 1442 bytes} [Fri Sep 23 13:43:11 2022] PUT /identity/v3/projects/feaf606da67b4ccf9411691df09f4216/groups/0ac89e094fe44bf999aa9f30f613fecd/roles/5ef6f90ed37e44adafc451b081a501d1 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:43:12.300217 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc8010d-6c73-468c-b945-b0a2c6509a66 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:12.350597 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:12.438732 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc8010d-6c73-468c-b945-b0a2c6509a66 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:12.480194 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a75150d-d586-4c52-be10-f0d8d92ad5dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:12.481195 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a75150d-d586-4c52-be10-f0d8d92ad5dd None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:12.481745 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a75150d-d586-4c52-be10-f0d8d92ad5dd None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:12.482223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a75150d-d586-4c52-be10-f0d8d92ad5dd None admin] PATH_INFO: `/v3/roles/member` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:12.495314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a75150d-d586-4c52-be10-f0d8d92ad5dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:12.501044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-3a75150d-d586-4c52-be10-f0d8d92ad5dd None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:43:12.501044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:12.501044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:12.501044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:12.501044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:12.501044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:12.501044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:12.501044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:12.501044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:12.501044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:12.501044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:12.501044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:12.501044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:12.501044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:12.501044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:12.501044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:12.501910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:12.501910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:12.501910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:12.501910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:12.501910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:12.501910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:12.501910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:12.501910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:12.501910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:12.501910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:12.501910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:12.501910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:12.501910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:12.501910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:12.501910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:12.501910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:12.502805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:12.502805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:12.502805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:12.502805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:12.502805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:12.502805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:12.502805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:12.502805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:12.502805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:12.502805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:12.502805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:43:12.502805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:12.505478 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 193/390] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:12 2022] GET /identity/v3/roles/member => generated 84 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:12.515477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9542c66-8b16-4c37-9cfd-547953c135fe None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:12.670248 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9542c66-8b16-4c37-9cfd-547953c135fe None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:12.713397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e1ad84-9d14-499a-b7bc-9c82be70ef71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:12.714237 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e1ad84-9d14-499a-b7bc-9c82be70ef71 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:12.714369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e1ad84-9d14-499a-b7bc-9c82be70ef71 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:12.714578 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e1ad84-9d14-499a-b7bc-9c82be70ef71 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:12.715244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e1ad84-9d14-499a-b7bc-9c82be70ef71 None admin] RBAC: Adding query filter params (name=member) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:12.715543 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e1ad84-9d14-499a-b7bc-9c82be70ef71 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:12.716896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e1ad84-9d14-499a-b7bc-9c82be70ef71 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:12.725919 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 198/391] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:43:12 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:12.732546 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a75150d-d586-4c52-be10-f0d8d92ad5dd None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:12.733760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932aea55-199e-4765-8f6b-75886e1b28a6 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:12.734047 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932aea55-199e-4765-8f6b-75886e1b28a6 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:12.734317 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932aea55-199e-4765-8f6b-75886e1b28a6 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:12.735661 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 199/392] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:12.749487 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4327de61-e9a4-40da-82e9-81bde1078281 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:12.749808 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4327de61-e9a4-40da-82e9-81bde1078281 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:12.750025 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4327de61-e9a4-40da-82e9-81bde1078281 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:12.770175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-4327de61-e9a4-40da-82e9-81bde1078281 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:12.771448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-4327de61-e9a4-40da-82e9-81bde1078281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:12.783306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:12.861604 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 200/393] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:12.869059 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:12.877284 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a75150d-d586-4c52-be10-f0d8d92ad5dd None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:12.916116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac784215-c39f-48c2-8157-f21d7950cde4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:12.917040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac784215-c39f-48c2-8157-f21d7950cde4 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:12.917481 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac784215-c39f-48c2-8157-f21d7950cde4 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:12.917993 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac784215-c39f-48c2-8157-f21d7950cde4 None admin] PATH_INFO: `/v3/groups/0ac89e094fe44bf999aa9f30f613fecd` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:12.919298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac784215-c39f-48c2-8157-f21d7950cde4 None admin] RBAC: Authorizing `identity:get_group(group_id=0ac89e094fe44bf999aa9f30f613fecd)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:12.920918 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac784215-c39f-48c2-8157-f21d7950cde4 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:12.922990 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 194/394] 10.222.0.62 () {56 vars in 1180 bytes} [Fri Sep 23 13:43:12 2022] GET /identity/v3/groups/0ac89e094fe44bf999aa9f30f613fecd => generated 226 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:12.928803 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4327de61-e9a4-40da-82e9-81bde1078281 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:13.055251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4327de61-e9a4-40da-82e9-81bde1078281 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:13.099238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4613feff-838a-48d6-986d-5b2f0b53f5b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:13.100186 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4613feff-838a-48d6-986d-5b2f0b53f5b5 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:13.100620 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4613feff-838a-48d6-986d-5b2f0b53f5b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:13.101035 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4613feff-838a-48d6-986d-5b2f0b53f5b5 None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:13.103214 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4613feff-838a-48d6-986d-5b2f0b53f5b5 None admin] RBAC: Authorizing `identity:get_project(project_id=feaf606da67b4ccf9411691df09f4216)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:13.104914 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4613feff-838a-48d6-986d-5b2f0b53f5b5 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:13.107268 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 201/395] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:43:12 2022] GET /identity/v3/projects/feaf606da67b4ccf9411691df09f4216 => generated 298 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:13.113059 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac784215-c39f-48c2-8157-f21d7950cde4 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:13.240663 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac784215-c39f-48c2-8157-f21d7950cde4 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:13.258195 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4613feff-838a-48d6-986d-5b2f0b53f5b5 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:13.280458 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:13.308899 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6cfd75a-bffb-40c6-bc4f-0f39f658cf64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:13.310164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cfd75a-bffb-40c6-bc4f-0f39f658cf64 None admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:13.311908 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cfd75a-bffb-40c6-bc4f-0f39f658cf64 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:13.312349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6cfd75a-bffb-40c6-bc4f-0f39f658cf64 None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216/groups/0ac89e094fe44bf999aa9f30f613fecd/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:13.314500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cfd75a-bffb-40c6-bc4f-0f39f658cf64 None admin] RBAC: Authorizing `identity:create_grant(project_id=feaf606da67b4ccf9411691df09f4216, group_id=0ac89e094fe44bf999aa9f30f613fecd, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:13.316763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6cfd75a-bffb-40c6-bc4f-0f39f658cf64 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:13.386950 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4613feff-838a-48d6-986d-5b2f0b53f5b5 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:13.397235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 195/396] 10.222.0.62 () {58 vars in 1442 bytes} [Fri Sep 23 13:43:13 2022] PUT /identity/v3/projects/feaf606da67b4ccf9411691df09f4216/groups/0ac89e094fe44bf999aa9f30f613fecd/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:43:13.398221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96c9e6e-30df-4fc7-9ad9-a93631859dd4 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:13.398523 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96c9e6e-30df-4fc7-9ad9-a93631859dd4 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:13.398788 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96c9e6e-30df-4fc7-9ad9-a93631859dd4 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:13.406686 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 196/397] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:13 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:13.411927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6617ea6-15d3-4c10-a3be-b68c4e513fc7 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:13.412208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6617ea6-15d3-4c10-a3be-b68c4e513fc7 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:13.412876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6617ea6-15d3-4c10-a3be-b68c4e513fc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:13.439921 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8622ed2-9f10-49f0-948d-1d76c8ca547a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:13.440922 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8622ed2-9f10-49f0-948d-1d76c8ca547a None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:13.441404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8622ed2-9f10-49f0-948d-1d76c8ca547a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:13.441878 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8622ed2-9f10-49f0-948d-1d76c8ca547a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:13.450692 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-b6617ea6-15d3-4c10-a3be-b68c4e513fc7 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:13.452015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-b6617ea6-15d3-4c10-a3be-b68c4e513fc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:13.455210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8622ed2-9f10-49f0-948d-1d76c8ca547a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:13.461581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-f8622ed2-9f10-49f0-948d-1d76c8ca547a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:43:13.461581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:13.461581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:13.461581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:13.461581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:13.461581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:13.461581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:13.461581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:13.461581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:13.461581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:13.461581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:13.461581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:13.461581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:13.461581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:13.461581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:13.461581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:13.462835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:13.462835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:13.462835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:13.462835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:13.462835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:13.462835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:13.462835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:13.462835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:13.462835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:13.462835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:13.462835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:13.462835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:13.462835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:13.462835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:13.462835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:13.462835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:13.463734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:13.463734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:13.463734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:13.463734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:13.463734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:13.463734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:13.463734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:13.463734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:13.463734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:13.463734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:13.463734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:43:13.463734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:13.471044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 202/398] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:13 2022] GET /identity/v3/roles/member => generated 84 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:43:13.473717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8622ed2-9f10-49f0-948d-1d76c8ca547a None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:13.489098 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:13.572829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:13.607794 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8622ed2-9f10-49f0-948d-1d76c8ca547a None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:13.610726 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 197/399] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:13.666687 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fe6cc0b-9386-41dc-ac95-10e82a952838 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:13.667309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe6cc0b-9386-41dc-ac95-10e82a952838 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:13.667585 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe6cc0b-9386-41dc-ac95-10e82a952838 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:13.667852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fe6cc0b-9386-41dc-ac95-10e82a952838 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:13.668589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe6cc0b-9386-41dc-ac95-10e82a952838 None admin] RBAC: Adding query filter params (name=member) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:13.668891 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe6cc0b-9386-41dc-ac95-10e82a952838 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:13.670281 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fe6cc0b-9386-41dc-ac95-10e82a952838 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:13.683613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 203/400] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:43:13 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:13.689539 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6617ea6-15d3-4c10-a3be-b68c4e513fc7 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:13.710593 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d37eac8-85ac-4ae5-b337-482256ad4658 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:13.710871 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d37eac8-85ac-4ae5-b337-482256ad4658 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:13.711137 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d37eac8-85ac-4ae5-b337-482256ad4658 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:13.713180 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 204/401] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:43:13.718438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1862901d-cb99-4247-a09c-c8e24fb8b6ae None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:13.718982 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1862901d-cb99-4247-a09c-c8e24fb8b6ae None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:13.718982 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1862901d-cb99-4247-a09c-c8e24fb8b6ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:13.744600 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-1862901d-cb99-4247-a09c-c8e24fb8b6ae None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:13.744600 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-1862901d-cb99-4247-a09c-c8e24fb8b6ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:13.825849 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 205/402] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:13.826110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6617ea6-15d3-4c10-a3be-b68c4e513fc7 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:13.886969 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae494d86-8b65-48e8-a0c3-d1a96e631419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:13.887741 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae494d86-8b65-48e8-a0c3-d1a96e631419 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:13.888016 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae494d86-8b65-48e8-a0c3-d1a96e631419 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:13.888281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae494d86-8b65-48e8-a0c3-d1a96e631419 None admin] PATH_INFO: `/v3/groups/0ac89e094fe44bf999aa9f30f613fecd` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:13.889447 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae494d86-8b65-48e8-a0c3-d1a96e631419 None admin] RBAC: Authorizing `identity:get_group(group_id=0ac89e094fe44bf999aa9f30f613fecd)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:13.890881 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae494d86-8b65-48e8-a0c3-d1a96e631419 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:13.892748 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 198/403] 10.222.0.62 () {56 vars in 1180 bytes} [Fri Sep 23 13:43:13 2022] GET /identity/v3/groups/0ac89e094fe44bf999aa9f30f613fecd => generated 226 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:13.898259 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1862901d-cb99-4247-a09c-c8e24fb8b6ae None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:13.916581 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae494d86-8b65-48e8-a0c3-d1a96e631419 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:13.958501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:14.014589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:14.063420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1862901d-cb99-4247-a09c-c8e24fb8b6ae None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:14.097299 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae494d86-8b65-48e8-a0c3-d1a96e631419 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:14.125429 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26c3af0-d744-4eb0-ab62-4cbfa454816e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:14.126066 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26c3af0-d744-4eb0-ab62-4cbfa454816e None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:14.126379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26c3af0-d744-4eb0-ab62-4cbfa454816e None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:14.126636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26c3af0-d744-4eb0-ab62-4cbfa454816e None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:14.127832 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26c3af0-d744-4eb0-ab62-4cbfa454816e None admin] RBAC: Authorizing `identity:get_project(project_id=8d4ba5b452824beab548b1646270d12c)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:14.129305 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26c3af0-d744-4eb0-ab62-4cbfa454816e None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:14.131293 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 206/404] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:43:13 2022] GET /identity/v3/projects/8d4ba5b452824beab548b1646270d12c => generated 302 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:14.137015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26c3af0-d744-4eb0-ab62-4cbfa454816e None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:14.168604 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cde393-7799-43e6-853d-a1b827a5d0e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:14.169602 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cde393-7799-43e6-853d-a1b827a5d0e9 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:14.170191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cde393-7799-43e6-853d-a1b827a5d0e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:14.170680 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cde393-7799-43e6-853d-a1b827a5d0e9 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:14.187734 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cde393-7799-43e6-853d-a1b827a5d0e9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:14.193088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-a4cde393-7799-43e6-853d-a1b827a5d0e9 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:43:14.193088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:14.193088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:14.193088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:14.193088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:14.193088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:14.193088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:14.193088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:14.193088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:14.193088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:14.193088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:14.193088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:14.193088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:14.193088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:14.193088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:14.193088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:14.193949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:14.193949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:14.193949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:14.193949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:14.193949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:14.193949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:14.193949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:14.193949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:14.193949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:14.193949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:14.193949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:14.193949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:14.193949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:14.193949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:14.193949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:14.193949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:14.194752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:14.194752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:14.194752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:14.194752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:14.194752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:14.194752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:14.194752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:14.194752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:14.194752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:14.194752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:14.194752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:43:14.194752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:14.197284 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 199/405] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:43:13 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:14.198226 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cde393-7799-43e6-853d-a1b827a5d0e9 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:14.284392 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26c3af0-d744-4eb0-ab62-4cbfa454816e None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:14.311199 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cde393-7799-43e6-853d-a1b827a5d0e9 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:14.322689 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:14.332539 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705fc06c-eb14-479d-b8ae-c3421070f2d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:14.333863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705fc06c-eb14-479d-b8ae-c3421070f2d4 None admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:14.334413 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705fc06c-eb14-479d-b8ae-c3421070f2d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:14.334733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705fc06c-eb14-479d-b8ae-c3421070f2d4 None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c/groups/0ac89e094fe44bf999aa9f30f613fecd/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:14.337909 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705fc06c-eb14-479d-b8ae-c3421070f2d4 None admin] RBAC: Authorizing `identity:create_grant(project_id=8d4ba5b452824beab548b1646270d12c, group_id=0ac89e094fe44bf999aa9f30f613fecd, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:14.339373 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705fc06c-eb14-479d-b8ae-c3421070f2d4 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:14.342014 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:14.351963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3066a0d9-b5f2-4846-840f-15a003e2c16e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:14.353409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3066a0d9-b5f2-4846-840f-15a003e2c16e None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:14.353699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3066a0d9-b5f2-4846-840f-15a003e2c16e None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:14.354076 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3066a0d9-b5f2-4846-840f-15a003e2c16e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:14.364301 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3066a0d9-b5f2-4846-840f-15a003e2c16e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:14.370264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-3066a0d9-b5f2-4846-840f-15a003e2c16e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:43:14.370264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:14.370264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:14.370264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:14.370264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:14.370264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:14.370264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:14.370264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:14.370264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:14.370264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:14.370264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:14.370264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:14.370264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:14.370264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:14.370264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:14.370264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:14.372710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:14.372710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:14.372710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:14.372710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:14.372710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:14.372710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:14.372710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:14.372710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:14.372710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:14.372710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:14.372710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:14.372710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:14.372710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:14.372710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:14.372710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:14.372710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:14.373701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:14.373701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:14.373701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:14.373701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:14.373701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:14.373701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:14.373701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:14.373701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:14.373701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:14.373701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:14.373701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:43:14.373701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:14.378587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 200/406] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:43:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:14.379417 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3066a0d9-b5f2-4846-840f-15a003e2c16e None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:14.454132 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 207/407] 10.222.0.62 () {58 vars in 1442 bytes} [Fri Sep 23 13:43:14 2022] PUT /identity/v3/projects/8d4ba5b452824beab548b1646270d12c/groups/0ac89e094fe44bf999aa9f30f613fecd/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 13:43:14.456266 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705fc06c-eb14-479d-b8ae-c3421070f2d4 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:14.536750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3066a0d9-b5f2-4846-840f-15a003e2c16e None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:14.608694 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705fc06c-eb14-479d-b8ae-c3421070f2d4 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:14.636042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8267d505-fc4c-4be3-b331-bc7968e577e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:14.636666 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8267d505-fc4c-4be3-b331-bc7968e577e4 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:14.636942 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8267d505-fc4c-4be3-b331-bc7968e577e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:14.637213 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8267d505-fc4c-4be3-b331-bc7968e577e4 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:14.637894 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8267d505-fc4c-4be3-b331-bc7968e577e4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:14.638197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8267d505-fc4c-4be3-b331-bc7968e577e4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:14.639622 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8267d505-fc4c-4be3-b331-bc7968e577e4 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:14.654695 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dddc03e-de81-4fed-a5c1-ed59d795731e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:14.655314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dddc03e-de81-4fed-a5c1-ed59d795731e None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:14.655598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dddc03e-de81-4fed-a5c1-ed59d795731e None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:14.655846 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dddc03e-de81-4fed-a5c1-ed59d795731e None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:14.656523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dddc03e-de81-4fed-a5c1-ed59d795731e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:14.656820 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dddc03e-de81-4fed-a5c1-ed59d795731e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:14.657775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 201/408] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:14 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:14.658240 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dddc03e-de81-4fed-a5c1-ed59d795731e None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:14.665192 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8267d505-fc4c-4be3-b331-bc7968e577e4 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:14.667881 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 208/409] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:14.668919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:14.673350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dddc03e-de81-4fed-a5c1-ed59d795731e None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:14.733047 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:14.778931 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dddc03e-de81-4fed-a5c1-ed59d795731e None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:14.787653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8267d505-fc4c-4be3-b331-bc7968e577e4 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:14.814249 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1f186e-eb1a-448e-ab9e-b251c9999bec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:14.814919 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1f186e-eb1a-448e-ab9e-b251c9999bec None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:14.815198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1f186e-eb1a-448e-ab9e-b251c9999bec None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:14.815466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1f186e-eb1a-448e-ab9e-b251c9999bec None admin] PATH_INFO: `/v3/groups/023e6ecb4fb747e5ae691eb4cf72e500` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:14.816588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1f186e-eb1a-448e-ab9e-b251c9999bec None admin] RBAC: Authorizing `identity:get_group(group_id=023e6ecb4fb747e5ae691eb4cf72e500)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:14.818009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1f186e-eb1a-448e-ab9e-b251c9999bec None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:14.819900 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 209/410] 10.222.0.62 () {56 vars in 1180 bytes} [Fri Sep 23 13:43:14 2022] GET /identity/v3/groups/023e6ecb4fb747e5ae691eb4cf72e500 => generated 229 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:14.825445 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1f186e-eb1a-448e-ab9e-b251c9999bec None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:14.832004 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb159e6-ac8c-4b87-9cec-1822e991811a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:14.832004 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb159e6-ac8c-4b87-9cec-1822e991811a None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:14.832004 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb159e6-ac8c-4b87-9cec-1822e991811a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:14.832004 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb159e6-ac8c-4b87-9cec-1822e991811a None admin] PATH_INFO: `/v3/groups/0ac89e094fe44bf999aa9f30f613fecd` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:14.832344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb159e6-ac8c-4b87-9cec-1822e991811a None admin] RBAC: Authorizing `identity:get_group(group_id=0ac89e094fe44bf999aa9f30f613fecd)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:14.833766 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb159e6-ac8c-4b87-9cec-1822e991811a None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:14.835771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 202/411] 10.222.0.62 () {56 vars in 1180 bytes} [Fri Sep 23 13:43:14 2022] GET /identity/v3/groups/0ac89e094fe44bf999aa9f30f613fecd => generated 226 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:14.841197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb159e6-ac8c-4b87-9cec-1822e991811a None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:14.936851 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:14.963592 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb159e6-ac8c-4b87-9cec-1822e991811a None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:15.033859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c56beac-5320-43ed-99fd-8834ccc0856e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:15.034677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c56beac-5320-43ed-99fd-8834ccc0856e None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:15.035089 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c56beac-5320-43ed-99fd-8834ccc0856e None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:15.035453 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c56beac-5320-43ed-99fd-8834ccc0856e None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:15.037351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c56beac-5320-43ed-99fd-8834ccc0856e None admin] RBAC: Authorizing `identity:get_project(project_id=8d4ba5b452824beab548b1646270d12c)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:15.038982 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c56beac-5320-43ed-99fd-8834ccc0856e None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:15.044757 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 203/412] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:43:14 2022] GET /identity/v3/projects/8d4ba5b452824beab548b1646270d12c => generated 302 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:15.047949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c56beac-5320-43ed-99fd-8834ccc0856e None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:15.053306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1f186e-eb1a-448e-ab9e-b251c9999bec None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:15.106953 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:15.127029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d55af8f-d198-4373-910a-e8c80f990bd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:15.127686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d55af8f-d198-4373-910a-e8c80f990bd1 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:15.127972 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d55af8f-d198-4373-910a-e8c80f990bd1 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:15.128250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d55af8f-d198-4373-910a-e8c80f990bd1 None admin] PATH_INFO: `/v3/projects/e15d4d7f53c543e88cdf9075fbadb3c8` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:15.129412 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d55af8f-d198-4373-910a-e8c80f990bd1 None admin] RBAC: Authorizing `identity:get_project(project_id=e15d4d7f53c543e88cdf9075fbadb3c8)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:15.130886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d55af8f-d198-4373-910a-e8c80f990bd1 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:15.132822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 210/413] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:43:14 2022] GET /identity/v3/projects/e15d4d7f53c543e88cdf9075fbadb3c8 => generated 344 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:15.143256 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d55af8f-d198-4373-910a-e8c80f990bd1 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:15.194121 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c56beac-5320-43ed-99fd-8834ccc0856e None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:15.234174 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b5340c-6dda-4cf7-a731-29fc3fa3096e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:15.235261 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b5340c-6dda-4cf7-a731-29fc3fa3096e None admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:15.235987 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b5340c-6dda-4cf7-a731-29fc3fa3096e None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:15.236458 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b5340c-6dda-4cf7-a731-29fc3fa3096e None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c/groups/0ac89e094fe44bf999aa9f30f613fecd/roles/5ef6f90ed37e44adafc451b081a501d1` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:15.239521 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b5340c-6dda-4cf7-a731-29fc3fa3096e None admin] RBAC: Authorizing `identity:create_grant(project_id=8d4ba5b452824beab548b1646270d12c, group_id=0ac89e094fe44bf999aa9f30f613fecd, role_id=5ef6f90ed37e44adafc451b081a501d1)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:15.242587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b5340c-6dda-4cf7-a731-29fc3fa3096e None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:15.325876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:15.331733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 204/414] 10.222.0.62 () {58 vars in 1442 bytes} [Fri Sep 23 13:43:15 2022] PUT /identity/v3/projects/8d4ba5b452824beab548b1646270d12c/groups/0ac89e094fe44bf999aa9f30f613fecd/roles/5ef6f90ed37e44adafc451b081a501d1 => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:43:15.337949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d55af8f-d198-4373-910a-e8c80f990bd1 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:15.382666 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df08026-d70a-437b-9927-38a379351635 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:15.384140 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288aa7f0-3c89-46ee-af19-bf7b556c4974 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:15.384294 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df08026-d70a-437b-9927-38a379351635 None admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:15.384560 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df08026-d70a-437b-9927-38a379351635 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:15.384917 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df08026-d70a-437b-9927-38a379351635 None admin] PATH_INFO: `/v3/projects/e15d4d7f53c543e88cdf9075fbadb3c8/groups/023e6ecb4fb747e5ae691eb4cf72e500/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:15.385214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288aa7f0-3c89-46ee-af19-bf7b556c4974 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:15.385631 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288aa7f0-3c89-46ee-af19-bf7b556c4974 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:15.387077 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df08026-d70a-437b-9927-38a379351635 None admin] RBAC: Authorizing `identity:create_grant(project_id=e15d4d7f53c543e88cdf9075fbadb3c8, group_id=023e6ecb4fb747e5ae691eb4cf72e500, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:15.387631 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 205/415] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:15 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:43:15.390351 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df08026-d70a-437b-9927-38a379351635 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:15.393293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19116b5-d7c2-49bb-bb92-6c42bf9554f6 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:15.393729 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19116b5-d7c2-49bb-bb92-6c42bf9554f6 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:15.394133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19116b5-d7c2-49bb-bb92-6c42bf9554f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:15.416044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-d19116b5-d7c2-49bb-bb92-6c42bf9554f6 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:15.417768 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d19116b5-d7c2-49bb-bb92-6c42bf9554f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:15.505981 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 206/416] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 23 13:43:15.514047 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 211/417] 10.222.0.62 () {58 vars in 1442 bytes} [Fri Sep 23 13:43:15 2022] PUT /identity/v3/projects/e15d4d7f53c543e88cdf9075fbadb3c8/groups/023e6ecb4fb747e5ae691eb4cf72e500/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:43:15.798016 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19116b5-d7c2-49bb-bb92-6c42bf9554f6 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:15.905587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19116b5-d7c2-49bb-bb92-6c42bf9554f6 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:15.947672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:15.957257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d7cc48-7bd4-43b4-9db4-774f9d30cf06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:15.958253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d7cc48-7bd4-43b4-9db4-774f9d30cf06 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:15.966964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d7cc48-7bd4-43b4-9db4-774f9d30cf06 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:15.966964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d7cc48-7bd4-43b4-9db4-774f9d30cf06 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:15.970691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d7cc48-7bd4-43b4-9db4-774f9d30cf06 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:15.975740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-18d7cc48-7bd4-43b4-9db4-774f9d30cf06 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:43:15.975740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:15.975740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:15.975740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:15.975740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:15.975740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:15.975740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:15.975740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:15.975740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:15.975740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:15.975740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:15.975740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:15.975740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:15.975740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:15.975740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:15.975740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:15.976839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:15.976839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:15.976839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:15.976839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:15.976839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:15.976839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:15.976839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:15.976839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:15.976839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:15.976839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:15.976839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:15.976839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:15.976839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:15.976839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:15.976839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:15.976839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:15.977859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:15.977859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:15.977859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:15.977859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:15.977859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:15.977859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:15.977859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:15.977859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:15.977859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:15.977859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:15.977859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:43:15.977859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:15.980221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 207/418] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:43:15 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:15.986160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df08026-d70a-437b-9927-38a379351635 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:16.058101 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:16.110535 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df08026-d70a-437b-9927-38a379351635 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:16.152773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2773e3d-18da-48f2-a4ae-c233c929b663 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:16.153072 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2773e3d-18da-48f2-a4ae-c233c929b663 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:16.153335 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2773e3d-18da-48f2-a4ae-c233c929b663 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:16.155760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 208/419] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:16.162399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de445c2e-a576-4bb9-8293-3716b34b21b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:16.162399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de445c2e-a576-4bb9-8293-3716b34b21b4 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:16.162610 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b5ac43-6760-4178-bb39-966d04921b9a None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:16.162700 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de445c2e-a576-4bb9-8293-3716b34b21b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:16.162782 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b5ac43-6760-4178-bb39-966d04921b9a None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:16.162850 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de445c2e-a576-4bb9-8293-3716b34b21b4 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:16.162919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b5ac43-6760-4178-bb39-966d04921b9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:16.162983 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de445c2e-a576-4bb9-8293-3716b34b21b4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:16.162983 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de445c2e-a576-4bb9-8293-3716b34b21b4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:16.163602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de445c2e-a576-4bb9-8293-3716b34b21b4 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:16.185026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 212/420] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:15 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:16.187893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de445c2e-a576-4bb9-8293-3716b34b21b4 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:16.195647 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-60b5ac43-6760-4178-bb39-966d04921b9a None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:16.197106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-60b5ac43-6760-4178-bb39-966d04921b9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:16.313586 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de445c2e-a576-4bb9-8293-3716b34b21b4 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:16.332972 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:16.348804 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 209/421] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 23 13:43:16.361728 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c471bb-eabf-4152-837a-501a0f6627b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:16.362734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c471bb-eabf-4152-837a-501a0f6627b6 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:16.363208 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c471bb-eabf-4152-837a-501a0f6627b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:16.363669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c471bb-eabf-4152-837a-501a0f6627b6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:16.440828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c471bb-eabf-4152-837a-501a0f6627b6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:16.442654 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c471bb-eabf-4152-837a-501a0f6627b6 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:16.544128 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 213/422] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:16.551502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b5ac43-6760-4178-bb39-966d04921b9a None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:16.718299 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b5ac43-6760-4178-bb39-966d04921b9a None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:16.720743 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c471bb-eabf-4152-837a-501a0f6627b6 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:16.738517 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:16.754543 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:16.788348 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288ac138-58e7-45a4-8a0d-e34938df0d21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:16.788985 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288ac138-58e7-45a4-8a0d-e34938df0d21 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:16.789257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288ac138-58e7-45a4-8a0d-e34938df0d21 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:16.789548 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288ac138-58e7-45a4-8a0d-e34938df0d21 None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:16.795053 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288ac138-58e7-45a4-8a0d-e34938df0d21 None admin] RBAC: Authorizing `identity:get_project(project_id=feaf606da67b4ccf9411691df09f4216)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:16.806102 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288ac138-58e7-45a4-8a0d-e34938df0d21 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:16.806102 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 210/423] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:43:16 2022] GET /identity/v3/projects/feaf606da67b4ccf9411691df09f4216 => generated 298 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:16.806102 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288ac138-58e7-45a4-8a0d-e34938df0d21 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:16.835289 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c471bb-eabf-4152-837a-501a0f6627b6 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:16.873035 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37348ab4-50b6-4c3c-aac5-7c0bc7566008 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:16.874022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37348ab4-50b6-4c3c-aac5-7c0bc7566008 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:16.874537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37348ab4-50b6-4c3c-aac5-7c0bc7566008 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:16.875018 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37348ab4-50b6-4c3c-aac5-7c0bc7566008 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:16.886909 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37348ab4-50b6-4c3c-aac5-7c0bc7566008 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:16.892104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-37348ab4-50b6-4c3c-aac5-7c0bc7566008 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:43:16.892104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:16.892104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:16.892104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:16.892104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:16.892104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:16.892104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:16.892104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:16.892104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:16.892104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:16.892104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:16.892104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:16.892104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:16.892104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:16.892104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:16.892104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:16.892952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:16.892952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:16.892952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:16.892952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:16.892952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:16.892952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:16.892952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:16.892952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:16.892952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:16.892952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:16.892952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:16.892952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:16.892952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:16.892952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:16.892952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:16.892952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:16.893751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:16.893751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:16.893751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:16.893751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:16.893751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:16.893751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:16.893751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:16.893751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:16.893751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:16.893751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:16.893751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:43:16.893751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:16.902367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 214/424] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:16 2022] GET /identity/v3/roles/member => generated 84 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:16.903893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37348ab4-50b6-4c3c-aac5-7c0bc7566008 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:16.986493 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:17.014434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37348ab4-50b6-4c3c-aac5-7c0bc7566008 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:17.057701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2defec-6fd8-484b-98db-729eb996dd96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:17.058729 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2defec-6fd8-484b-98db-729eb996dd96 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:17.059213 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2defec-6fd8-484b-98db-729eb996dd96 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:17.059671 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2defec-6fd8-484b-98db-729eb996dd96 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:17.060533 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2defec-6fd8-484b-98db-729eb996dd96 None admin] RBAC: Adding query filter params (name=member) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:17.061024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2defec-6fd8-484b-98db-729eb996dd96 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:17.062618 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2defec-6fd8-484b-98db-729eb996dd96 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:17.067780 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288ac138-58e7-45a4-8a0d-e34938df0d21 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:17.081197 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 215/425] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:43:16 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:17.090780 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2defec-6fd8-484b-98db-729eb996dd96 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:17.139454 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cd1360-b7f2-4fe2-b197-0f5217f9429b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:17.140504 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cd1360-b7f2-4fe2-b197-0f5217f9429b None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:17.140997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cd1360-b7f2-4fe2-b197-0f5217f9429b None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:17.141458 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cd1360-b7f2-4fe2-b197-0f5217f9429b None admin] PATH_INFO: `/v3/groups/0ac89e094fe44bf999aa9f30f613fecd` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:17.142970 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cd1360-b7f2-4fe2-b197-0f5217f9429b None admin] RBAC: Authorizing `identity:get_group(group_id=0ac89e094fe44bf999aa9f30f613fecd)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:17.144603 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21cd1360-b7f2-4fe2-b197-0f5217f9429b None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:17.148372 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 211/426] 10.222.0.62 () {56 vars in 1180 bytes} [Fri Sep 23 13:43:16 2022] GET /identity/v3/groups/0ac89e094fe44bf999aa9f30f613fecd => generated 226 bytes in 345 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:17.154586 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cd1360-b7f2-4fe2-b197-0f5217f9429b None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:17.271612 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:17.276569 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2defec-6fd8-484b-98db-729eb996dd96 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:17.331328 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21cd1360-b7f2-4fe2-b197-0f5217f9429b None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:17.347711 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f149f00f-0eec-400b-b645-0e44d78f8d41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:17.356791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f149f00f-0eec-400b-b645-0e44d78f8d41 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:17.357077 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f149f00f-0eec-400b-b645-0e44d78f8d41 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:17.357366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f149f00f-0eec-400b-b645-0e44d78f8d41 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:17.362028 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:17.373010 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c94949-ca3e-4d76-ab19-ff3fa6f89372 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:17.375156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c94949-ca3e-4d76-ab19-ff3fa6f89372 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:17.375156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c94949-ca3e-4d76-ab19-ff3fa6f89372 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:17.375156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c94949-ca3e-4d76-ab19-ff3fa6f89372 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:17.375570 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c94949-ca3e-4d76-ab19-ff3fa6f89372 None admin] RBAC: Adding query filter params (group.id=0ac89e094fe44bf999aa9f30f613fecd, role.id=5ef6f90ed37e44adafc451b081a501d1, scope.project.id=feaf606da67b4ccf9411691df09f4216) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:17.375886 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c94949-ca3e-4d76-ab19-ff3fa6f89372 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:17.377563 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c94949-ca3e-4d76-ab19-ff3fa6f89372 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:17.390527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 212/427] 10.222.0.62 () {56 vars in 1376 bytes} [Fri Sep 23 13:43:17 2022] GET /identity/v3/role_assignments?group.id=0ac89e094fe44bf999aa9f30f613fecd&scope.project.id=feaf606da67b4ccf9411691df09f4216&role.id=5ef6f90ed37e44adafc451b081a501d1 => generated 611 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:17.392503 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea71d13-3d1e-4b69-a5b3-972496b9d100 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:17.394297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea71d13-3d1e-4b69-a5b3-972496b9d100 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:17.395505 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea71d13-3d1e-4b69-a5b3-972496b9d100 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:17.398389 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 213/428] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:17 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:17.410205 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab6e25f-54cd-4aa6-9114-c15724b2e451 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:17.410730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab6e25f-54cd-4aa6-9114-c15724b2e451 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:17.411139 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab6e25f-54cd-4aa6-9114-c15724b2e451 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:17.439153 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-7ab6e25f-54cd-4aa6-9114-c15724b2e451 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:17.442629 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-7ab6e25f-54cd-4aa6-9114-c15724b2e451 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:17.484487 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f149f00f-0eec-400b-b645-0e44d78f8d41 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:17.489271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f149f00f-0eec-400b-b645-0e44d78f8d41 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:17.589764 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 214/429] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 23 13:43:17.649446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 216/430] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 559 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 23 13:43:17.655253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab6e25f-54cd-4aa6-9114-c15724b2e451 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:17.710725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:17.763382 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab6e25f-54cd-4aa6-9114-c15724b2e451 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:17.801570 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6e5f1d-f6fd-4703-8004-f1696507fe46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:17.802203 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6e5f1d-f6fd-4703-8004-f1696507fe46 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:17.802527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6e5f1d-f6fd-4703-8004-f1696507fe46 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:17.802802 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6e5f1d-f6fd-4703-8004-f1696507fe46 None admin] PATH_INFO: `/v3/projects/feaf606da67b4ccf9411691df09f4216` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:17.803933 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6e5f1d-f6fd-4703-8004-f1696507fe46 None admin] RBAC: Authorizing `identity:get_project(project_id=feaf606da67b4ccf9411691df09f4216)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:17.805681 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6e5f1d-f6fd-4703-8004-f1696507fe46 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:17.807949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 215/431] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:43:17 2022] GET /identity/v3/projects/feaf606da67b4ccf9411691df09f4216 => generated 298 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:17.813398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f149f00f-0eec-400b-b645-0e44d78f8d41 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:17.850072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:17.917064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6e5f1d-f6fd-4703-8004-f1696507fe46 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:17.927708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f149f00f-0eec-400b-b645-0e44d78f8d41 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:17.973554 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062dc0a6-72f8-419b-8f23-0502f9826e31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:17.974671 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062dc0a6-72f8-419b-8f23-0502f9826e31 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:17.975149 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062dc0a6-72f8-419b-8f23-0502f9826e31 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:17.975623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062dc0a6-72f8-419b-8f23-0502f9826e31 None admin] PATH_INFO: `/v3/groups/0ac89e094fe44bf999aa9f30f613fecd` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:17.977696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062dc0a6-72f8-419b-8f23-0502f9826e31 None admin] RBAC: Authorizing `identity:get_group(group_id=0ac89e094fe44bf999aa9f30f613fecd)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:17.979772 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062dc0a6-72f8-419b-8f23-0502f9826e31 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:17.982197 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 217/432] 10.222.0.62 () {56 vars in 1180 bytes} [Fri Sep 23 13:43:17 2022] GET /identity/v3/groups/0ac89e094fe44bf999aa9f30f613fecd => generated 226 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:17.989821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062dc0a6-72f8-419b-8f23-0502f9826e31 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:18.049959 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6e5f1d-f6fd-4703-8004-f1696507fe46 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:18.066988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:18.084791 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3726c7c-0e83-49ee-b739-ecfec6595245 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:18.085394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3726c7c-0e83-49ee-b739-ecfec6595245 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:18.085668 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3726c7c-0e83-49ee-b739-ecfec6595245 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:18.085934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3726c7c-0e83-49ee-b739-ecfec6595245 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:18.097611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3726c7c-0e83-49ee-b739-ecfec6595245 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:18.098791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062dc0a6-72f8-419b-8f23-0502f9826e31 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:18.101142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:18.103517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-f3726c7c-0e83-49ee-b739-ecfec6595245 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:43:18.103517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:18.103517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:18.103517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:18.103517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:18.103517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:18.103517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:18.103517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:18.103517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:18.103517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:18.103517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:18.103517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:18.103517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:18.103517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:18.103517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:18.103517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:18.104599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:18.104599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:18.104599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:18.104599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:18.104599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:18.104599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:18.104599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:18.104599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:18.104599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:18.104599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:18.104599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:18.104599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:18.104599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:18.104599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:18.104599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:18.104599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:18.105425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:18.105425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:18.105425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:18.105425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:18.105425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:18.105425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:18.105425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:18.105425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:18.105425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:18.105425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:18.105425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 23 13:43:18.105425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:18.105425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 216/433] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:18.107188 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964dd2d4-62b5-44a3-bc89-86143d28ee5c None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:18.107476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964dd2d4-62b5-44a3-bc89-86143d28ee5c None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:18.107749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964dd2d4-62b5-44a3-bc89-86143d28ee5c None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:18.113063 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 217/434] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:18 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:18.114179 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb6e040-e6b8-4b2f-bba8-dad1f5b4f55d None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:18.114362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb6e040-e6b8-4b2f-bba8-dad1f5b4f55d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:18.114642 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb6e040-e6b8-4b2f-bba8-dad1f5b4f55d None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:18.115840 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 218/435] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:18 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:18.116521 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb6e040-e6b8-4b2f-bba8-dad1f5b4f55d None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:18.135836 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d2e161-071f-47c4-b41e-490939dca8e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:18.136441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d2e161-071f-47c4-b41e-490939dca8e7 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:18.136721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d2e161-071f-47c4-b41e-490939dca8e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:18.136995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d2e161-071f-47c4-b41e-490939dca8e7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:18.137718 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d2e161-071f-47c4-b41e-490939dca8e7 None admin] RBAC: Adding query filter params (group.id=0ac89e094fe44bf999aa9f30f613fecd, role.id=e8367770d3124f2db1821b6af66bfeab, scope.project.id=feaf606da67b4ccf9411691df09f4216) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:18.138027 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d2e161-071f-47c4-b41e-490939dca8e7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:18.139537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d2e161-071f-47c4-b41e-490939dca8e7 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:18.145311 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 218/436] 10.222.0.62 () {56 vars in 1376 bytes} [Fri Sep 23 13:43:17 2022] GET /identity/v3/role_assignments?group.id=0ac89e094fe44bf999aa9f30f613fecd&scope.project.id=feaf606da67b4ccf9411691df09f4216&role.id=e8367770d3124f2db1821b6af66bfeab => generated 611 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:18.146466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059453aa-26d0-448b-ad7c-7f57d23a4315 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:18.146754 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059453aa-26d0-448b-ad7c-7f57d23a4315 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:18.147027 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059453aa-26d0-448b-ad7c-7f57d23a4315 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:18.169109 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-059453aa-26d0-448b-ad7c-7f57d23a4315 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:18.170421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-059453aa-26d0-448b-ad7c-7f57d23a4315 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:18.240668 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb6e040-e6b8-4b2f-bba8-dad1f5b4f55d None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:18.254522 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 219/437] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:18.256365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17eef7df-bed2-495b-b4d5-6968e9611a18 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:18.256833 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17eef7df-bed2-495b-b4d5-6968e9611a18 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:18.257326 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17eef7df-bed2-495b-b4d5-6968e9611a18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:18.286617 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-17eef7df-bed2-495b-b4d5-6968e9611a18 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:18.287875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-17eef7df-bed2-495b-b4d5-6968e9611a18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:18.290615 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0189307b-fa46-4e06-aa13-0fa6731a9d86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:18.291628 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0189307b-fa46-4e06-aa13-0fa6731a9d86 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:18.291970 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0189307b-fa46-4e06-aa13-0fa6731a9d86 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:18.292311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0189307b-fa46-4e06-aa13-0fa6731a9d86 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:18.293055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0189307b-fa46-4e06-aa13-0fa6731a9d86 None admin] RBAC: Adding query filter params (name=member) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:18.293418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0189307b-fa46-4e06-aa13-0fa6731a9d86 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:18.294976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0189307b-fa46-4e06-aa13-0fa6731a9d86 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:18.309488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 219/438] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:43:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:18.310784 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0189307b-fa46-4e06-aa13-0fa6731a9d86 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:18.364904 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:18.378928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 220/439] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:18.434977 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0189307b-fa46-4e06-aa13-0fa6731a9d86 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:18.470537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0760c9-9e0b-43cc-9282-3e324b8a7e96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:18.471140 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0760c9-9e0b-43cc-9282-3e324b8a7e96 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:18.471419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0760c9-9e0b-43cc-9282-3e324b8a7e96 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:18.471743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0760c9-9e0b-43cc-9282-3e324b8a7e96 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:18.545765 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0760c9-9e0b-43cc-9282-3e324b8a7e96 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:18.545975 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17eef7df-bed2-495b-b4d5-6968e9611a18 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:18.547742 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0760c9-9e0b-43cc-9282-3e324b8a7e96 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:18.582546 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:18.588605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:18.672222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 220/440] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:18.672858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0760c9-9e0b-43cc-9282-3e324b8a7e96 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:18.689217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17eef7df-bed2-495b-b4d5-6968e9611a18 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:18.735412 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d5e747-f2a9-4f4a-ba54-3b4816265ea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:18.736036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d5e747-f2a9-4f4a-ba54-3b4816265ea9 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:18.736316 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d5e747-f2a9-4f4a-ba54-3b4816265ea9 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:18.736588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d5e747-f2a9-4f4a-ba54-3b4816265ea9 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:18.749795 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d5e747-f2a9-4f4a-ba54-3b4816265ea9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:18.755650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-71d5e747-f2a9-4f4a-ba54-3b4816265ea9 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:43:18.755650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:18.755650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:18.755650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:18.755650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:18.755650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:18.755650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:18.755650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:18.755650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:18.755650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:18.755650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:18.755650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:18.755650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:18.755650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:18.755650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:18.755650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:18.756708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:18.756708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:18.756708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:18.756708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:18.756708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:18.756708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:18.756708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:18.756708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:18.756708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:18.756708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:18.756708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:18.756708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:18.756708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:18.756708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:18.756708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:18.756708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:18.757531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:18.757531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:18.757531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:18.757531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:18.757531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:18.757531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:18.757531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:18.757531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:18.757531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:18.757531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:18.757531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:43:18.757531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:18.757531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 221/441] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:43:18 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:18.758218 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d5e747-f2a9-4f4a-ba54-3b4816265ea9 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:18.795696 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0760c9-9e0b-43cc-9282-3e324b8a7e96 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:18.834884 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142e57d4-be72-4cfe-a52d-2e7f19928156 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:18.835508 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142e57d4-be72-4cfe-a52d-2e7f19928156 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:18.835790 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142e57d4-be72-4cfe-a52d-2e7f19928156 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:18.836057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142e57d4-be72-4cfe-a52d-2e7f19928156 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:18.843774 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:18.847677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142e57d4-be72-4cfe-a52d-2e7f19928156 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:18.853167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-142e57d4-be72-4cfe-a52d-2e7f19928156 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:43:18.853167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:18.853167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:18.853167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:18.853167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:18.853167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:18.853167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:18.853167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:18.853167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:18.853167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:18.853167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:18.853167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:18.853167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:18.853167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:18.853167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:18.853167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:18.861052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:18.861052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:18.861052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:18.861052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:18.861052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:18.861052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:18.861052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:18.861052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:18.861052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:18.861052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:18.861052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:18.861052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:18.861052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:18.861052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:18.861052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:18.861052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:18.861824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:18.861824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:18.861824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:18.861824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:18.861824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:18.861824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:18.861824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:18.861824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:18.861824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:18.861824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:18.861824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:43:18.861824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:18.861824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 221/442] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:43:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:18.861824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142e57d4-be72-4cfe-a52d-2e7f19928156 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:18.862564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:18.871845 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d5e747-f2a9-4f4a-ba54-3b4816265ea9 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:18.905724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77c70a4-2435-431f-867f-e5b058c7f251 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:18.906367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77c70a4-2435-431f-867f-e5b058c7f251 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:18.906649 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77c70a4-2435-431f-867f-e5b058c7f251 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:18.906918 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e77c70a4-2435-431f-867f-e5b058c7f251 None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:18.908039 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77c70a4-2435-431f-867f-e5b058c7f251 None admin] RBAC: Authorizing `identity:get_project(project_id=8d4ba5b452824beab548b1646270d12c)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:18.909457 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e77c70a4-2435-431f-867f-e5b058c7f251 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:18.911098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 222/443] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:43:18 2022] GET /identity/v3/projects/8d4ba5b452824beab548b1646270d12c => generated 302 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:18.911853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77c70a4-2435-431f-867f-e5b058c7f251 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:18.954249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142e57d4-be72-4cfe-a52d-2e7f19928156 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:18.988542 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58570fe-6c5c-4366-b566-26f6a1f4d861 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:18.989142 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58570fe-6c5c-4366-b566-26f6a1f4d861 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:18.989425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58570fe-6c5c-4366-b566-26f6a1f4d861 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:18.989685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58570fe-6c5c-4366-b566-26f6a1f4d861 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:18.990360 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58570fe-6c5c-4366-b566-26f6a1f4d861 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:18.990664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58570fe-6c5c-4366-b566-26f6a1f4d861 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:18.992031 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58570fe-6c5c-4366-b566-26f6a1f4d861 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:19.000947 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 222/444] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:18 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:19.001723 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58570fe-6c5c-4366-b566-26f6a1f4d861 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:19.012599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e77c70a4-2435-431f-867f-e5b058c7f251 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:19.045313 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd477d1-a6c5-48a5-b0ae-3cfb8c435ed7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:19.045956 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd477d1-a6c5-48a5-b0ae-3cfb8c435ed7 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:19.046230 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd477d1-a6c5-48a5-b0ae-3cfb8c435ed7 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:19.046526 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd477d1-a6c5-48a5-b0ae-3cfb8c435ed7 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:19.047185 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd477d1-a6c5-48a5-b0ae-3cfb8c435ed7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:19.047479 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd477d1-a6c5-48a5-b0ae-3cfb8c435ed7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:19.048827 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd477d1-a6c5-48a5-b0ae-3cfb8c435ed7 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:19.057169 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 223/445] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:19.057952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd477d1-a6c5-48a5-b0ae-3cfb8c435ed7 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:19.065125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:19.076519 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:19.107332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58570fe-6c5c-4366-b566-26f6a1f4d861 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:19.141966 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8632f1f2-c7bb-42c7-850d-18c06c2d87c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:19.142611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8632f1f2-c7bb-42c7-850d-18c06c2d87c5 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:19.142888 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8632f1f2-c7bb-42c7-850d-18c06c2d87c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:19.143155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8632f1f2-c7bb-42c7-850d-18c06c2d87c5 None admin] PATH_INFO: `/v3/groups/0ac89e094fe44bf999aa9f30f613fecd` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:19.144267 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8632f1f2-c7bb-42c7-850d-18c06c2d87c5 None admin] RBAC: Authorizing `identity:get_group(group_id=0ac89e094fe44bf999aa9f30f613fecd)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:19.145687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8632f1f2-c7bb-42c7-850d-18c06c2d87c5 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:19.147274 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 223/446] 10.222.0.62 () {56 vars in 1180 bytes} [Fri Sep 23 13:43:19 2022] GET /identity/v3/groups/0ac89e094fe44bf999aa9f30f613fecd => generated 226 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:19.148024 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8632f1f2-c7bb-42c7-850d-18c06c2d87c5 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:19.161837 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd477d1-a6c5-48a5-b0ae-3cfb8c435ed7 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:19.197128 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad58189-e568-493b-b5ce-e3f9b0a606fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:19.197729 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad58189-e568-493b-b5ce-e3f9b0a606fd None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:19.198010 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad58189-e568-493b-b5ce-e3f9b0a606fd None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:19.198279 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad58189-e568-493b-b5ce-e3f9b0a606fd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:19.247540 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8632f1f2-c7bb-42c7-850d-18c06c2d87c5 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:19.272073 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad58189-e568-493b-b5ce-e3f9b0a606fd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:19.273517 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad58189-e568-493b-b5ce-e3f9b0a606fd None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:19.273688 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:19.281192 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43110cd3-91a1-4629-9ac4-b27864f8f4a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:19.281787 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43110cd3-91a1-4629-9ac4-b27864f8f4a5 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:19.282069 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43110cd3-91a1-4629-9ac4-b27864f8f4a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:19.282353 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43110cd3-91a1-4629-9ac4-b27864f8f4a5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:19.291027 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:19.349815 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43110cd3-91a1-4629-9ac4-b27864f8f4a5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:19.351585 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43110cd3-91a1-4629-9ac4-b27864f8f4a5 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:19.372846 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 224/447] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:19.373675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad58189-e568-493b-b5ce-e3f9b0a606fd None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:19.460690 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 224/448] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:19.461653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43110cd3-91a1-4629-9ac4-b27864f8f4a5 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:19.484136 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad58189-e568-493b-b5ce-e3f9b0a606fd None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:19.490599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:19.502561 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:19.517997 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebaf058-0032-41c7-9001-21da80104f95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:19.518634 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebaf058-0032-41c7-9001-21da80104f95 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:19.518910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebaf058-0032-41c7-9001-21da80104f95 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:19.519175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebaf058-0032-41c7-9001-21da80104f95 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:19.519888 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebaf058-0032-41c7-9001-21da80104f95 None admin] RBAC: Adding query filter params (group.id=0ac89e094fe44bf999aa9f30f613fecd, role.id=e8367770d3124f2db1821b6af66bfeab, scope.project.id=8d4ba5b452824beab548b1646270d12c) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:19.520195 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebaf058-0032-41c7-9001-21da80104f95 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:19.521662 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebaf058-0032-41c7-9001-21da80104f95 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:19.528226 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 225/449] 10.222.0.62 () {56 vars in 1376 bytes} [Fri Sep 23 13:43:19 2022] GET /identity/v3/role_assignments?group.id=0ac89e094fe44bf999aa9f30f613fecd&scope.project.id=8d4ba5b452824beab548b1646270d12c&role.id=e8367770d3124f2db1821b6af66bfeab => generated 611 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:19.528997 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebaf058-0032-41c7-9001-21da80104f95 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:19.568035 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43110cd3-91a1-4629-9ac4-b27864f8f4a5 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:19.615916 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3608ee-c764-49a9-b90d-e27681e3d657 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:19.616557 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3608ee-c764-49a9-b90d-e27681e3d657 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:19.616832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3608ee-c764-49a9-b90d-e27681e3d657 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:19.617095 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3608ee-c764-49a9-b90d-e27681e3d657 None admin] PATH_INFO: `/v3/projects/8d4ba5b452824beab548b1646270d12c` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:19.618530 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3608ee-c764-49a9-b90d-e27681e3d657 None admin] RBAC: Authorizing `identity:get_project(project_id=8d4ba5b452824beab548b1646270d12c)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:19.621018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3608ee-c764-49a9-b90d-e27681e3d657 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:19.622813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 225/450] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:43:19 2022] GET /identity/v3/projects/8d4ba5b452824beab548b1646270d12c => generated 302 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:19.628115 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3608ee-c764-49a9-b90d-e27681e3d657 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:19.654889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebaf058-0032-41c7-9001-21da80104f95 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:19.691119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18285632-e7d9-4e33-a230-bb9a13c675e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:19.691756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18285632-e7d9-4e33-a230-bb9a13c675e5 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:19.692054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18285632-e7d9-4e33-a230-bb9a13c675e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:19.692322 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18285632-e7d9-4e33-a230-bb9a13c675e5 None admin] PATH_INFO: `/v3/projects/e15d4d7f53c543e88cdf9075fbadb3c8` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:19.693481 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18285632-e7d9-4e33-a230-bb9a13c675e5 None admin] RBAC: Authorizing `identity:get_project(project_id=e15d4d7f53c543e88cdf9075fbadb3c8)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:19.694994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18285632-e7d9-4e33-a230-bb9a13c675e5 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:19.696907 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 226/451] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:43:19 2022] GET /identity/v3/projects/e15d4d7f53c543e88cdf9075fbadb3c8 => generated 344 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:19.702362 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18285632-e7d9-4e33-a230-bb9a13c675e5 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:19.742856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:19.747025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3608ee-c764-49a9-b90d-e27681e3d657 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:19.772040 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:19.785101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4d2bad-76ba-41e9-bd8d-05c425af60b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:19.785731 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4d2bad-76ba-41e9-bd8d-05c425af60b9 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:19.786432 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4d2bad-76ba-41e9-bd8d-05c425af60b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:19.786578 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4d2bad-76ba-41e9-bd8d-05c425af60b9 None admin] PATH_INFO: `/v3/groups/0ac89e094fe44bf999aa9f30f613fecd` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:19.787641 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4d2bad-76ba-41e9-bd8d-05c425af60b9 None admin] RBAC: Authorizing `identity:get_group(group_id=0ac89e094fe44bf999aa9f30f613fecd)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:19.789086 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4d2bad-76ba-41e9-bd8d-05c425af60b9 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:19.791944 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 226/452] 10.222.0.62 () {56 vars in 1180 bytes} [Fri Sep 23 13:43:19 2022] GET /identity/v3/groups/0ac89e094fe44bf999aa9f30f613fecd => generated 226 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:19.798441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4d2bad-76ba-41e9-bd8d-05c425af60b9 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:19.837536 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18285632-e7d9-4e33-a230-bb9a13c675e5 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:19.879868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6472980d-2d63-495c-a964-623fd442bd3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:19.880919 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6472980d-2d63-495c-a964-623fd442bd3b None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:19.881199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6472980d-2d63-495c-a964-623fd442bd3b None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:19.881469 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6472980d-2d63-495c-a964-623fd442bd3b None admin] PATH_INFO: `/v3/groups/023e6ecb4fb747e5ae691eb4cf72e500` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:19.882657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6472980d-2d63-495c-a964-623fd442bd3b None admin] RBAC: Authorizing `identity:get_group(group_id=023e6ecb4fb747e5ae691eb4cf72e500)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:19.884152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6472980d-2d63-495c-a964-623fd442bd3b None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:19.885731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 227/453] 10.222.0.62 () {56 vars in 1180 bytes} [Fri Sep 23 13:43:19 2022] GET /identity/v3/groups/023e6ecb4fb747e5ae691eb4cf72e500 => generated 229 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:19.891765 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6472980d-2d63-495c-a964-623fd442bd3b None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:19.909908 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4d2bad-76ba-41e9-bd8d-05c425af60b9 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:19.950750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82597873-c333-43a8-a13b-046c75699e63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:19.951351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82597873-c333-43a8-a13b-046c75699e63 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:19.951635 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82597873-c333-43a8-a13b-046c75699e63 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:19.951904 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82597873-c333-43a8-a13b-046c75699e63 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:19.952626 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82597873-c333-43a8-a13b-046c75699e63 None admin] RBAC: Adding query filter params (group.id=0ac89e094fe44bf999aa9f30f613fecd, role.id=5ef6f90ed37e44adafc451b081a501d1, scope.project.id=8d4ba5b452824beab548b1646270d12c) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:19.952932 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82597873-c333-43a8-a13b-046c75699e63 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:19.954427 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82597873-c333-43a8-a13b-046c75699e63 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:19.960077 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 227/454] 10.222.0.62 () {56 vars in 1376 bytes} [Fri Sep 23 13:43:19 2022] GET /identity/v3/role_assignments?group.id=0ac89e094fe44bf999aa9f30f613fecd&scope.project.id=8d4ba5b452824beab548b1646270d12c&role.id=5ef6f90ed37e44adafc451b081a501d1 => generated 611 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:20.015553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6472980d-2d63-495c-a964-623fd442bd3b None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:20.041763 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:20.059798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf09b829-6617-43d9-8dbe-c4beb78e87b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:20.060419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf09b829-6617-43d9-8dbe-c4beb78e87b0 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:20.060701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf09b829-6617-43d9-8dbe-c4beb78e87b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:20.060975 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf09b829-6617-43d9-8dbe-c4beb78e87b0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:20.061723 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf09b829-6617-43d9-8dbe-c4beb78e87b0 None admin] RBAC: Adding query filter params (group.id=023e6ecb4fb747e5ae691eb4cf72e500, role.id=727f0fa598e54b9a929d2cf6277c875e, scope.project.id=e15d4d7f53c543e88cdf9075fbadb3c8) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:20.062035 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf09b829-6617-43d9-8dbe-c4beb78e87b0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:20.063537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf09b829-6617-43d9-8dbe-c4beb78e87b0 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:20.069886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 228/455] 10.222.0.62 () {56 vars in 1376 bytes} [Fri Sep 23 13:43:19 2022] GET /identity/v3/role_assignments?group.id=023e6ecb4fb747e5ae691eb4cf72e500&scope.project.id=e15d4d7f53c543e88cdf9075fbadb3c8&role.id=727f0fa598e54b9a929d2cf6277c875e => generated 611 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:22.687777 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b93805e-5b3f-414a-b311-e6fb3becd17a None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:22.688253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b93805e-5b3f-414a-b311-e6fb3becd17a None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:22.688253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b93805e-5b3f-414a-b311-e6fb3becd17a None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:22.689861 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 228/456] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:22.694770 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccd41f1-908d-4180-85a4-d6db451265d9 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:22.695198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccd41f1-908d-4180-85a4-d6db451265d9 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:22.695954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccd41f1-908d-4180-85a4-d6db451265d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:22.719330 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-9ccd41f1-908d-4180-85a4-d6db451265d9 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:22.720603 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9ccd41f1-908d-4180-85a4-d6db451265d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:22.815955 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 229/457] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:22.958198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65ff288-24ca-4351-b4f6-88b125e75d9d None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:22.958751 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65ff288-24ca-4351-b4f6-88b125e75d9d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:22.959169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65ff288-24ca-4351-b4f6-88b125e75d9d None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:22.961128 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 229/458] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:22.966826 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608a7d79-2f86-47ff-8d41-946ba568760c None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:22.967477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608a7d79-2f86-47ff-8d41-946ba568760c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:22.967947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608a7d79-2f86-47ff-8d41-946ba568760c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:23.003673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-608a7d79-2f86-47ff-8d41-946ba568760c None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:23.005281 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-608a7d79-2f86-47ff-8d41-946ba568760c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:23.099929 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 230/459] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 23 13:43:23.109508 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47eb328-c84f-4a3e-a0e4-36b99907145f None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:23.110027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47eb328-c84f-4a3e-a0e4-36b99907145f None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:23.110520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47eb328-c84f-4a3e-a0e4-36b99907145f None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:23.112058 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 230/460] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:23.117809 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af43984-894b-42d2-b872-fc19050d39e3 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:23.118349 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af43984-894b-42d2-b872-fc19050d39e3 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:23.118824 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af43984-894b-42d2-b872-fc19050d39e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:23.147291 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-0af43984-894b-42d2-b872-fc19050d39e3 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:23.148817 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-0af43984-894b-42d2-b872-fc19050d39e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:23.190669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:23.195936 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47eb328-c84f-4a3e-a0e4-36b99907145f None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:23.226597 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 231/461] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:23.263703 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:23.376983 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47eb328-c84f-4a3e-a0e4-36b99907145f None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:23.428140 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72491f3-e0de-4553-a41d-4ec6f4e2f22e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:23.428779 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72491f3-e0de-4553-a41d-4ec6f4e2f22e None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:23.429058 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72491f3-e0de-4553-a41d-4ec6f4e2f22e None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:23.429329 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72491f3-e0de-4553-a41d-4ec6f4e2f22e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:23.444930 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72491f3-e0de-4553-a41d-4ec6f4e2f22e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:23.447591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b72491f3-e0de-4553-a41d-4ec6f4e2f22e None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:23.466511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af43984-894b-42d2-b872-fc19050d39e3 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:23.467219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-b72491f3-e0de-4553-a41d-4ec6f4e2f22e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:23.467219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:23.467219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:23.467219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:23.467219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:23.467219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:23.467219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:23.467219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:23.467219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:23.467219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:23.467219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:23.467219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:23.467219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:23.467219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:23.467219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:23.467219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:23.468112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:23.468112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:23.468112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:23.468112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:23.468112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:23.468112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:23.468112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:23.468112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:23.468112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:23.468112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:23.468112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:23.468112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:23.468112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:23.468112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:23.468112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:23.468112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:23.468112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:23.468961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:23.468961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:23.468961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:23.468961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 231/462] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:23.474918 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72491f3-e0de-4553-a41d-4ec6f4e2f22e None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:23.591885 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af43984-894b-42d2-b872-fc19050d39e3 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:23.659049 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:23.675121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed62c65f-ea95-4969-90c7-09bd99f51522 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:23.676080 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed62c65f-ea95-4969-90c7-09bd99f51522 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:23.677972 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed62c65f-ea95-4969-90c7-09bd99f51522 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:23.677972 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed62c65f-ea95-4969-90c7-09bd99f51522 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:23.688958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72491f3-e0de-4553-a41d-4ec6f4e2f22e None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:23.690901 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed62c65f-ea95-4969-90c7-09bd99f51522 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:23.692834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed62c65f-ea95-4969-90c7-09bd99f51522 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:23.713474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-ed62c65f-ea95-4969-90c7-09bd99f51522 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:23.713474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:23.713474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:23.713474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:23.713474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:23.713474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:23.713474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:23.713474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:23.713474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:23.713474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:23.713474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:23.713474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:23.713474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:23.713474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:23.713474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:23.713474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:23.714949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:23.714949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:23.714949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:23.714949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:23.714949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:23.714949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:23.714949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:23.714949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:23.714949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:23.714949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:23.714949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:23.714949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:23.714949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:23.714949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:23.714949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:23.714949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:23.714949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:23.715791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:23.715791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:23.715791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:23.718092 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 232/463] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:23.719111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed62c65f-ea95-4969-90c7-09bd99f51522 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:23.733665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc626c8d-bd04-40f2-9383-05bbfd04529b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:23.734246 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc626c8d-bd04-40f2-9383-05bbfd04529b None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:23.734560 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc626c8d-bd04-40f2-9383-05bbfd04529b None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:23.734824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc626c8d-bd04-40f2-9383-05bbfd04529b None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:23.735461 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc626c8d-bd04-40f2-9383-05bbfd04529b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:23.735764 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc626c8d-bd04-40f2-9383-05bbfd04529b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:23.737109 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc626c8d-bd04-40f2-9383-05bbfd04529b None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:23.750473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:23.756492 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 232/464] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:23.757557 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc626c8d-bd04-40f2-9383-05bbfd04529b None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:23.834018 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed62c65f-ea95-4969-90c7-09bd99f51522 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:23.861542 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc626c8d-bd04-40f2-9383-05bbfd04529b None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:23.872305 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee42f88-a392-4a12-a095-2c6979b01824 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:23.873254 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee42f88-a392-4a12-a095-2c6979b01824 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:23.873680 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee42f88-a392-4a12-a095-2c6979b01824 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:23.873958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee42f88-a392-4a12-a095-2c6979b01824 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:23.884657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee42f88-a392-4a12-a095-2c6979b01824 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:23.886099 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee42f88-a392-4a12-a095-2c6979b01824 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:23.896752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f157257-0292-43b9-93f7-afbd3643782f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:23.897359 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f157257-0292-43b9-93f7-afbd3643782f None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:23.897637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f157257-0292-43b9-93f7-afbd3643782f None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:23.897907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f157257-0292-43b9-93f7-afbd3643782f None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:23.898891 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f157257-0292-43b9-93f7-afbd3643782f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:23.899070 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f157257-0292-43b9-93f7-afbd3643782f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:23.900394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f157257-0292-43b9-93f7-afbd3643782f None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:23.901085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-5ee42f88-a392-4a12-a095-2c6979b01824 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:23.901085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:23.901085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:23.901085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:23.901085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:23.901085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:23.901085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:23.901085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:23.901085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:23.901085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:23.901085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:23.901085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:23.901085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:23.901085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:23.901085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:23.901085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:23.901947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:23.901947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:23.901947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:23.901947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:23.901947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:23.901947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:23.901947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:23.901947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:23.901947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:23.901947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:23.901947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:23.901947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:23.901947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:23.901947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:23.901947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:23.901947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:23.901947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:23.902829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:23.902829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:23.902829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:23.902829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 233/465] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:23.903597 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee42f88-a392-4a12-a095-2c6979b01824 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:23.916304 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:23.919777 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 233/466] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:23.920848 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cd1074-3f8a-4596-a0f3-4e06ce7e9916 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:23.921125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cd1074-3f8a-4596-a0f3-4e06ce7e9916 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:23.921396 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cd1074-3f8a-4596-a0f3-4e06ce7e9916 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:23.922684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 234/467] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:23 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:23.927649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cd1074-3f8a-4596-a0f3-4e06ce7e9916 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:23.995158 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:24.018152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee42f88-a392-4a12-a095-2c6979b01824 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:24.039502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cd1074-3f8a-4596-a0f3-4e06ce7e9916 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:24.055112 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b51526-1342-410c-a57d-4ef38266f203 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:24.055112 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b51526-1342-410c-a57d-4ef38266f203 None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:24.056111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b51526-1342-410c-a57d-4ef38266f203 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:24.056183 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b51526-1342-410c-a57d-4ef38266f203 None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:24.056818 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b51526-1342-410c-a57d-4ef38266f203 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:24.058406 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b51526-1342-410c-a57d-4ef38266f203 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:24.076903 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa12a522-2d7d-41f9-9b96-45a1a60ff9c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:24.085357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa12a522-2d7d-41f9-9b96-45a1a60ff9c8 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:24.085357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa12a522-2d7d-41f9-9b96-45a1a60ff9c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:24.085357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa12a522-2d7d-41f9-9b96-45a1a60ff9c8 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:24.085357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa12a522-2d7d-41f9-9b96-45a1a60ff9c8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:24.085357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa12a522-2d7d-41f9-9b96-45a1a60ff9c8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:24.085357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa12a522-2d7d-41f9-9b96-45a1a60ff9c8 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:24.092889 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 235/468] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:24.093971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa12a522-2d7d-41f9-9b96-45a1a60ff9c8 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:24.152507 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:24.192414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa12a522-2d7d-41f9-9b96-45a1a60ff9c8 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:24.225199 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac349390-7d71-4228-8bd8-fe21a6f320b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:24.225797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac349390-7d71-4228-8bd8-fe21a6f320b8 None admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:24.226074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac349390-7d71-4228-8bd8-fe21a6f320b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:24.226444 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac349390-7d71-4228-8bd8-fe21a6f320b8 None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:24.227198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac349390-7d71-4228-8bd8-fe21a6f320b8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:24.228613 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac349390-7d71-4228-8bd8-fe21a6f320b8 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:24.260200 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 234/469] 10.222.0.62 () {60 vars in 1132 bytes} [Fri Sep 23 13:43:23 2022] POST /identity/v3/users => generated 246 bytes in 357 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:43:24.261364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4d3182-f39d-480d-8408-6587ee2c836a None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:24.261651 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4d3182-f39d-480d-8408-6587ee2c836a None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:24.261921 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4d3182-f39d-480d-8408-6587ee2c836a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:24.281817 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-db4d3182-f39d-480d-8408-6587ee2c836a None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:24.283091 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-db4d3182-f39d-480d-8408-6587ee2c836a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:24.346232 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 236/470] 10.222.0.62 () {60 vars in 1132 bytes} [Fri Sep 23 13:43:24 2022] POST /identity/v3/users => generated 248 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:43:24.347549 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac349390-7d71-4228-8bd8-fe21a6f320b8 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:24.368292 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 235/471] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:24.453042 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d99614-3d96-441b-b302-cb56e4062491 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:24.453524 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d99614-3d96-441b-b302-cb56e4062491 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:24.454194 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d99614-3d96-441b-b302-cb56e4062491 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:24.456047 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 236/472] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:24 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:43:24.462563 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138625ce-2bf8-49a1-8e1b-dd29d079cfa9 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:24.462990 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138625ce-2bf8-49a1-8e1b-dd29d079cfa9 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:24.463370 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138625ce-2bf8-49a1-8e1b-dd29d079cfa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:24.493719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-138625ce-2bf8-49a1-8e1b-dd29d079cfa9 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:24.496919 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-138625ce-2bf8-49a1-8e1b-dd29d079cfa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:24.513783 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac349390-7d71-4228-8bd8-fe21a6f320b8 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:24.534129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:24.557028 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e32d335-4a34-42aa-bed3-1d336616912c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:24.557748 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e32d335-4a34-42aa-bed3-1d336616912c None admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:24.558092 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e32d335-4a34-42aa-bed3-1d336616912c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:24.558473 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e32d335-4a34-42aa-bed3-1d336616912c None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:24.559224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e32d335-4a34-42aa-bed3-1d336616912c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:24.560915 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e32d335-4a34-42aa-bed3-1d336616912c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:24.572380 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:24.586134 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 237/473] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:24.660069 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 237/474] 10.222.0.62 () {60 vars in 1132 bytes} [Fri Sep 23 13:43:24 2022] POST /identity/v3/users => generated 248 bytes in 313 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:43:24.714511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138625ce-2bf8-49a1-8e1b-dd29d079cfa9 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:24.848217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138625ce-2bf8-49a1-8e1b-dd29d079cfa9 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:24.897666 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5590e024-7eeb-40a6-8d7a-0aafacb04da0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:24.899179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5590e024-7eeb-40a6-8d7a-0aafacb04da0 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:24.899613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5590e024-7eeb-40a6-8d7a-0aafacb04da0 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:24.900018 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5590e024-7eeb-40a6-8d7a-0aafacb04da0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:24.915493 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5590e024-7eeb-40a6-8d7a-0aafacb04da0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:24.917179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5590e024-7eeb-40a6-8d7a-0aafacb04da0 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:24.935734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-5590e024-7eeb-40a6-8d7a-0aafacb04da0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:24.935734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:24.935734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:24.935734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:24.935734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:24.935734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:24.935734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:24.935734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:24.935734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:24.935734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:24.935734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:24.935734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:24.935734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:24.935734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:24.935734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:24.935734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:24.936848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:24.936848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:24.936848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:24.936848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:24.936848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:24.936848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:24.936848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:24.936848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:24.936848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:24.936848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:24.936848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:24.936848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:24.936848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:24.936848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:24.936848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:24.936848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:24.936848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:24.937891 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:24.937891 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:24.937891 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:24.939620 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 238/475] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:24.945881 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e32d335-4a34-42aa-bed3-1d336616912c None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:25.002570 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5590e024-7eeb-40a6-8d7a-0aafacb04da0 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:25.060103 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:25.084112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e32d335-4a34-42aa-bed3-1d336616912c None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:25.148624 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5590e024-7eeb-40a6-8d7a-0aafacb04da0 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:25.172361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9d0ce2-7e92-4871-b311-ad1d62e3fd26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:25.173369 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9d0ce2-7e92-4871-b311-ad1d62e3fd26 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:25.173841 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9d0ce2-7e92-4871-b311-ad1d62e3fd26 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:25.174292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9d0ce2-7e92-4871-b311-ad1d62e3fd26 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:25.175150 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9d0ce2-7e92-4871-b311-ad1d62e3fd26 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:25.175661 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9d0ce2-7e92-4871-b311-ad1d62e3fd26 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:25.177579 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9d0ce2-7e92-4871-b311-ad1d62e3fd26 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:25.199719 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 238/476] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:25.205698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9d0ce2-7e92-4871-b311-ad1d62e3fd26 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:25.233067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b36269f-b776-42ad-8783-294812f8b0e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:25.233755 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b36269f-b776-42ad-8783-294812f8b0e9 None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:25.234039 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b36269f-b776-42ad-8783-294812f8b0e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:25.234365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b36269f-b776-42ad-8783-294812f8b0e9 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:25.235056 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b36269f-b776-42ad-8783-294812f8b0e9 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:25.236434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b36269f-b776-42ad-8783-294812f8b0e9 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:25.261840 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:25.283860 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.sql.core [None req-9b36269f-b776-42ad-8783-294812f8b0e9 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 23 13:43:25.283860 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 23 13:43:25.283860 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [parameters: {'id': 'ff000030a4dc4421a3ec7d584bfef4a7', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 23 13:43:25.283860 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81569) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 23 13:43:25.313530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-9b36269f-b776-42ad-8783-294812f8b0e9 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 23 13:43:25.313530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:25.313530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 23 13:43:25.313530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 23 13:43:25.313530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 23 13:43:25.313530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 23 13:43:25.313530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 23 13:43:25.313530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application result = self._query(query) Sep 23 13:43:25.313530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 23 13:43:25.313530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application conn.query(q) Sep 23 13:43:25.313530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 23 13:43:25.313530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 23 13:43:25.313530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 23 13:43:25.313530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application result.read() Sep 23 13:43:25.313530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 23 13:43:25.314474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 23 13:43:25.314474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 23 13:43:25.314474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 23 13:43:25.314474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 23 13:43:25.314474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 23 13:43:25.314474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 23 13:43:25.314474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 23 13:43:25.314474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 23 13:43:25.314474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:25.314474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 23 13:43:25.314474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:25.314474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:25.314474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 23 13:43:25.314474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 23 13:43:25.314474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 23 13:43:25.314474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return ref.to_dict() Sep 23 13:43:25.314474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 23 13:43:25.314474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application next(self.gen) Sep 23 13:43:25.315387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 23 13:43:25.315387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application yield resource Sep 23 13:43:25.315387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 23 13:43:25.315387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application next(self.gen) Sep 23 13:43:25.315387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 23 13:43:25.315387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self.session.rollback() Sep 23 13:43:25.315387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 23 13:43:25.315387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self.force_reraise() Sep 23 13:43:25.315387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 23 13:43:25.315387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise self.value Sep 23 13:43:25.315387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 23 13:43:25.315387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 23 13:43:25.315387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 23 13:43:25.315387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application session.commit() Sep 23 13:43:25.315387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 23 13:43:25.315387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 23 13:43:25.315387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 23 13:43:25.316242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._prepare_impl() Sep 23 13:43:25.316242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 23 13:43:25.316242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self.session.flush() Sep 23 13:43:25.316242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 23 13:43:25.316242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._flush(objects) Sep 23 13:43:25.316242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 23 13:43:25.316242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 23 13:43:25.316242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 23 13:43:25.316242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application compat.raise_( Sep 23 13:43:25.316242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 23 13:43:25.316242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception Sep 23 13:43:25.316242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 23 13:43:25.316242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application flush_context.execute() Sep 23 13:43:25.316242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 23 13:43:25.316242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rec.execute(self) Sep 23 13:43:25.316242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 23 13:43:25.316242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 23 13:43:25.317090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 23 13:43:25.317090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 23 13:43:25.317090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 23 13:43:25.317090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 23 13:43:25.317090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 23 13:43:25.317090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 23 13:43:25.317090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 23 13:43:25.317090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 23 13:43:25.317090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 23 13:43:25.317090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 23 13:43:25.317090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 23 13:43:25.317090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 23 13:43:25.317090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 23 13:43:25.317090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 23 13:43:25.317090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 23 13:43:25.317090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception Sep 23 13:43:25.317882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 23 13:43:25.317882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 23 13:43:25.317882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 23 13:43:25.317882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 23 13:43:25.317882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 23 13:43:25.317882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application result = self._query(query) Sep 23 13:43:25.317882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 23 13:43:25.317882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application conn.query(q) Sep 23 13:43:25.317882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 23 13:43:25.317882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 23 13:43:25.317882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 23 13:43:25.317882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application result.read() Sep 23 13:43:25.317882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 23 13:43:25.317882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 23 13:43:25.317882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 23 13:43:25.317882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 23 13:43:25.320306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 23 13:43:25.320306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 23 13:43:25.320306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 23 13:43:25.320306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 23 13:43:25.320306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 23 13:43:25.320306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 23 13:43:25.320306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [parameters: {'id': 'ff000030a4dc4421a3ec7d584bfef4a7', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 23 13:43:25.320306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 23 13:43:25.320306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:25.320306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:43:25.320306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:25.320306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:25.320306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:25.320306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:25.320306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:25.320306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:25.322282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:25.322282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:25.322282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:25.322282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:25.322282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:25.322282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:25.322282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 23 13:43:25.322282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 23 13:43:25.322282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:25.322282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:25.322282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 23 13:43:25.322282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 23 13:43:25.322282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 23 13:43:25.322282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 23 13:43:25.322282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 23 13:43:25.322282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:25.324278 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 239/477] 10.222.0.62 () {60 vars in 1131 bytes} [Fri Sep 23 13:43:25 2022] POST /identity/v3/roles => generated 144 bytes in 313 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:43:25.324278 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b36269f-b776-42ad-8783-294812f8b0e9 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:25.377055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9d0ce2-7e92-4871-b311-ad1d62e3fd26 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:25.443120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d20dcf-2788-43b8-a9c6-4f7ef33e4972 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:25.444745 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d20dcf-2788-43b8-a9c6-4f7ef33e4972 None admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:25.445063 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d20dcf-2788-43b8-a9c6-4f7ef33e4972 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:25.446010 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d20dcf-2788-43b8-a9c6-4f7ef33e4972 None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:25.450253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d20dcf-2788-43b8-a9c6-4f7ef33e4972 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:25.451762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d20dcf-2788-43b8-a9c6-4f7ef33e4972 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:25.530224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b36269f-b776-42ad-8783-294812f8b0e9 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:25.543570 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:25.546625 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 239/478] 10.222.0.62 () {60 vars in 1132 bytes} [Fri Sep 23 13:43:25 2022] POST /identity/v3/users => generated 249 bytes in 341 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:43:25.581832 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a356f3c7-1b6e-4a7c-ae48-6f61768d4c05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:25.582529 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a356f3c7-1b6e-4a7c-ae48-6f61768d4c05 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:25.582815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a356f3c7-1b6e-4a7c-ae48-6f61768d4c05 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:25.583237 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a356f3c7-1b6e-4a7c-ae48-6f61768d4c05 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:25.600963 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a356f3c7-1b6e-4a7c-ae48-6f61768d4c05 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:25.606614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-a356f3c7-1b6e-4a7c-ae48-6f61768d4c05 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:43:25.606614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:25.606614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:25.606614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:25.606614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:25.606614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:25.606614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:25.606614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:25.606614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:25.606614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:25.606614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:25.606614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:25.606614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:25.606614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:25.606614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:25.606614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:25.608657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:25.608657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:25.608657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:25.608657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:25.608657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:25.608657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:25.608657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:25.608657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:25.608657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:25.608657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:25.608657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:25.608657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:25.608657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:25.608657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:25.608657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:25.608657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:25.610613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:25.610613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:25.610613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:25.610613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:25.610613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:25.610613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:25.610613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:25.610613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:25.610613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:25.610613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:25.610613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 23 13:43:25.610613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:25.610613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 240/479] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:43:25 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 287 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:43:25.618358 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d20dcf-2788-43b8-a9c6-4f7ef33e4972 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:25.727893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:25.741962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d20dcf-2788-43b8-a9c6-4f7ef33e4972 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:25.787707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba73ebd2-158f-4d52-b213-23a71916f429 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:25.788332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba73ebd2-158f-4d52-b213-23a71916f429 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:25.788676 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba73ebd2-158f-4d52-b213-23a71916f429 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:25.789613 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba73ebd2-158f-4d52-b213-23a71916f429 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:25.790528 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba73ebd2-158f-4d52-b213-23a71916f429 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:25.791920 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba73ebd2-158f-4d52-b213-23a71916f429 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:25.793390 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba73ebd2-158f-4d52-b213-23a71916f429 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:25.802167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 240/480] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:25 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:26.955620 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28ae047-c5ef-4b20-b506-a394df9a619b None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:26.956306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28ae047-c5ef-4b20-b506-a394df9a619b None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:26.956306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28ae047-c5ef-4b20-b506-a394df9a619b None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:26.957766 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 241/481] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:26.963472 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9990a31-0bd6-473d-81c0-a0c0d005ede8 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:26.963963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9990a31-0bd6-473d-81c0-a0c0d005ede8 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:26.964111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9990a31-0bd6-473d-81c0-a0c0d005ede8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:26.991328 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-d9990a31-0bd6-473d-81c0-a0c0d005ede8 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:26.993340 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d9990a31-0bd6-473d-81c0-a0c0d005ede8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:27.096795 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 241/482] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:27.198466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68144cb8-d6d4-4c31-afa2-a841b701fd08 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:27.203712 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68144cb8-d6d4-4c31-afa2-a841b701fd08 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:27.203712 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68144cb8-d6d4-4c31-afa2-a841b701fd08 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:27.203712 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 242/483] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:27.216848 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac20a9b-e858-4706-9490-8642e48d7631 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:27.217132 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac20a9b-e858-4706-9490-8642e48d7631 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:27.217406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ac20a9b-e858-4706-9490-8642e48d7631 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:27.244939 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-1ac20a9b-e858-4706-9490-8642e48d7631 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:27.246438 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-1ac20a9b-e858-4706-9490-8642e48d7631 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:27.348526 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 242/484] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:27.406414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d45cbc5-a598-45d5-86cd-898ef550c291 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:27.406733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d45cbc5-a598-45d5-86cd-898ef550c291 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:27.406957 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d45cbc5-a598-45d5-86cd-898ef550c291 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:27.408431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 243/485] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:27.414196 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49fa317-4193-4ecb-8053-261a3a5ab9c0 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:27.414516 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49fa317-4193-4ecb-8053-261a3a5ab9c0 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:27.414789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49fa317-4193-4ecb-8053-261a3a5ab9c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:27.440855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c49fa317-4193-4ecb-8053-261a3a5ab9c0 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:27.442057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c49fa317-4193-4ecb-8053-261a3a5ab9c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:27.445671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:27.459052 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d45cbc5-a598-45d5-86cd-898ef550c291 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:27.551961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 243/486] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:27.640125 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d45cbc5-a598-45d5-86cd-898ef550c291 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:27.684566 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4037682-8ed7-4923-a307-3a0043616e12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:27.685191 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4037682-8ed7-4923-a307-3a0043616e12 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:27.685469 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4037682-8ed7-4923-a307-3a0043616e12 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:27.685742 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4037682-8ed7-4923-a307-3a0043616e12 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:27.707126 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4037682-8ed7-4923-a307-3a0043616e12 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:27.712444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-c4037682-8ed7-4923-a307-3a0043616e12 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:27.712444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:27.712444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:27.712444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:27.712444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:27.712444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:27.712444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:27.712444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:27.712444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:27.712444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:27.712444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:27.712444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:27.712444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:27.712444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:27.712444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:27.712444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:27.713304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:27.713304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:27.713304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:27.713304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:27.713304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:27.713304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:27.713304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:27.713304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:27.713304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:27.713304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:27.713304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:27.713304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:27.713304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:27.713304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:27.713304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:27.713304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:27.714084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:27.714084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:27.714084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:27.714084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:27.714084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:27.714084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:27.714084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:27.714084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:27.714084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:27.714084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:27.714084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:27.714084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:27.714084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 244/487] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:43:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:27.719451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49fa317-4193-4ecb-8053-261a3a5ab9c0 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:27.839683 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4037682-8ed7-4923-a307-3a0043616e12 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:27.859808 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49fa317-4193-4ecb-8053-261a3a5ab9c0 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:27.879973 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:27.896562 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8a45d2-903a-4660-98c8-e4d4af0c696d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:27.897177 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8a45d2-903a-4660-98c8-e4d4af0c696d None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:27.903246 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8a45d2-903a-4660-98c8-e4d4af0c696d None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:27.903246 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8a45d2-903a-4660-98c8-e4d4af0c696d None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:27.903246 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8a45d2-903a-4660-98c8-e4d4af0c696d None admin] RBAC: Adding query filter params (name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:27.903246 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8a45d2-903a-4660-98c8-e4d4af0c696d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:27.903246 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e8a45d2-903a-4660-98c8-e4d4af0c696d None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:27.912139 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 244/488] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:43:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:27.914126 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8a45d2-903a-4660-98c8-e4d4af0c696d None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:27.917913 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:27.975202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4037682-8ed7-4923-a307-3a0043616e12 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:28.021717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0342c455-5bb1-4525-81fb-0bccf21a07c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:28.022658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0342c455-5bb1-4525-81fb-0bccf21a07c7 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:28.023860 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0342c455-5bb1-4525-81fb-0bccf21a07c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:28.024288 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0342c455-5bb1-4525-81fb-0bccf21a07c7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:28.040116 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0342c455-5bb1-4525-81fb-0bccf21a07c7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:28.042478 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8a45d2-903a-4660-98c8-e4d4af0c696d None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:28.046778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-0342c455-5bb1-4525-81fb-0bccf21a07c7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:28.046778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:28.046778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:28.046778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:28.046778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:28.046778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:28.046778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:28.046778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:28.046778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:28.046778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:28.046778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:28.046778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:28.046778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:28.046778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:28.046778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:28.046778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:28.047828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:28.047828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:28.047828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:28.047828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:28.047828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:28.047828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:28.047828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:28.047828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:28.047828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:28.047828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:28.047828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:28.047828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:28.047828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:28.047828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:28.047828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:28.047828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:28.048778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:28.048778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:28.048778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:28.048778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:28.048778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:28.048778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:28.048778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:28.048778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:28.048778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:28.048778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:28.048778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:28.048778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:28.048778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 245/489] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:43:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:28.049433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0342c455-5bb1-4525-81fb-0bccf21a07c7 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:28.096730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3865f511-2522-4dc7-9a5f-793015cce673 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:28.097611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3865f511-2522-4dc7-9a5f-793015cce673 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:28.098031 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3865f511-2522-4dc7-9a5f-793015cce673 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:28.098550 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3865f511-2522-4dc7-9a5f-793015cce673 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:28.113036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3865f511-2522-4dc7-9a5f-793015cce673 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:28.121042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-3865f511-2522-4dc7-9a5f-793015cce673 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:28.121042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:28.121042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:28.121042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:28.121042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:28.121042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:28.121042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:28.121042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:28.121042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:28.121042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:28.121042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:28.121042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:28.121042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:28.121042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:28.121042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:28.121042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:28.122146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:28.122146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:28.122146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:28.122146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:28.122146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:28.122146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:28.122146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:28.122146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:28.122146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:28.122146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:28.122146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:28.122146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:28.122146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:28.122146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:28.122146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:28.122146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:28.123013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:28.123013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:28.123013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:28.123013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:28.123013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:28.123013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:28.123013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:28.123013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:28.123013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:28.123013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:28.123013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:28.123013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:28.123609 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 245/490] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:43:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:28.124351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3865f511-2522-4dc7-9a5f-793015cce673 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:28.174742 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:28.190879 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0342c455-5bb1-4525-81fb-0bccf21a07c7 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:28.202929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:28.228863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e1e0df-627a-48c5-a7b2-573c4c5278a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:28.229770 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e1e0df-627a-48c5-a7b2-573c4c5278a2 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:28.230240 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e1e0df-627a-48c5-a7b2-573c4c5278a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:28.230710 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e1e0df-627a-48c5-a7b2-573c4c5278a2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:28.241696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e1e0df-627a-48c5-a7b2-573c4c5278a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:28.243442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e1e0df-627a-48c5-a7b2-573c4c5278a2 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:28.246082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3865f511-2522-4dc7-9a5f-793015cce673 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:28.259979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-29e1e0df-627a-48c5-a7b2-573c4c5278a2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:28.259979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:28.259979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:28.259979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:28.259979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:28.259979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:28.259979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:28.259979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:28.259979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:28.259979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:28.259979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:28.259979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:28.259979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:28.259979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:28.259979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:28.259979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:28.260916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:28.260916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:28.260916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:28.260916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:28.260916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:28.260916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:28.260916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:28.260916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:28.260916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:28.260916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:28.260916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:28.260916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:28.260916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:28.260916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:28.260916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:28.260916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:28.260916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:28.261762 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:28.261762 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:28.261762 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:28.263673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 246/491] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:28.264705 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e1e0df-627a-48c5-a7b2-573c4c5278a2 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:28.285697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241e2ebe-77ac-45b9-9562-9464bd5fc47c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:28.286313 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241e2ebe-77ac-45b9-9562-9464bd5fc47c None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:28.286612 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241e2ebe-77ac-45b9-9562-9464bd5fc47c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:28.287043 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241e2ebe-77ac-45b9-9562-9464bd5fc47c None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:28.287849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241e2ebe-77ac-45b9-9562-9464bd5fc47c None admin] RBAC: Adding query filter params (name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:28.288289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241e2ebe-77ac-45b9-9562-9464bd5fc47c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:28.290876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241e2ebe-77ac-45b9-9562-9464bd5fc47c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:28.302361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 246/492] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:43:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:28.303340 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241e2ebe-77ac-45b9-9562-9464bd5fc47c None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:28.386894 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e1e0df-627a-48c5-a7b2-573c4c5278a2 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:28.427040 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ade7fe-9d09-43f0-9e4e-5359c5a417d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:28.428025 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ade7fe-9d09-43f0-9e4e-5359c5a417d0 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:28.428530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ade7fe-9d09-43f0-9e4e-5359c5a417d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:28.429075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ade7fe-9d09-43f0-9e4e-5359c5a417d0 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:28.431041 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ade7fe-9d09-43f0-9e4e-5359c5a417d0 None admin] RBAC: Adding query filter params (name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:28.431574 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ade7fe-9d09-43f0-9e4e-5359c5a417d0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:28.433208 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ade7fe-9d09-43f0-9e4e-5359c5a417d0 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:28.442995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 247/493] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:43:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:28.444302 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ade7fe-9d09-43f0-9e4e-5359c5a417d0 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:28.448334 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:28.452082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241e2ebe-77ac-45b9-9562-9464bd5fc47c None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:28.481816 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:28.493939 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b748a66c-1906-4c51-a922-a4badb41a3ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:28.494393 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b748a66c-1906-4c51-a922-a4badb41a3ab None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:28.494680 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b748a66c-1906-4c51-a922-a4badb41a3ab None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:28.494953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b748a66c-1906-4c51-a922-a4badb41a3ab None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:28.495900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b748a66c-1906-4c51-a922-a4badb41a3ab None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:28.496165 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b748a66c-1906-4c51-a922-a4badb41a3ab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:28.497542 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b748a66c-1906-4c51-a922-a4badb41a3ab None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:28.509763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 247/494] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:28.510808 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b748a66c-1906-4c51-a922-a4badb41a3ab None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:28.550347 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ade7fe-9d09-43f0-9e4e-5359c5a417d0 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:28.582939 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdec4391-ec2b-4c1d-87bb-776e7ab72050 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:28.583558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdec4391-ec2b-4c1d-87bb-776e7ab72050 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:28.583833 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdec4391-ec2b-4c1d-87bb-776e7ab72050 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:28.584100 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdec4391-ec2b-4c1d-87bb-776e7ab72050 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:28.595499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdec4391-ec2b-4c1d-87bb-776e7ab72050 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:28.596943 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdec4391-ec2b-4c1d-87bb-776e7ab72050 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:28.611893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-cdec4391-ec2b-4c1d-87bb-776e7ab72050 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:28.611893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:28.611893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:28.611893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:28.611893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:28.611893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:28.611893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:28.611893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:28.611893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:28.611893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:28.611893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:28.611893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:28.611893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:28.611893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:28.611893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:28.611893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:28.612910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:28.612910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:28.612910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:28.612910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:28.612910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:28.612910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:28.612910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:28.612910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:28.612910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:28.612910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:28.612910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:28.612910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:28.612910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:28.612910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:28.612910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:28.612910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:28.612910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:28.613762 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:28.613762 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:28.613762 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:28.613762 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 248/495] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:28.614356 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb91046-61e8-4af8-8d24-4d893eb2cc73 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:28.614452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb91046-61e8-4af8-8d24-4d893eb2cc73 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:28.614678 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb91046-61e8-4af8-8d24-4d893eb2cc73 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:28.614779 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b748a66c-1906-4c51-a922-a4badb41a3ab None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:28.616092 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 249/496] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:28 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:28.617231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb91046-61e8-4af8-8d24-4d893eb2cc73 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:28.651544 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff36370-d9ff-40ea-8fac-ae258781cc41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:28.652087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff36370-d9ff-40ea-8fac-ae258781cc41 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:28.652365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff36370-d9ff-40ea-8fac-ae258781cc41 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:28.652628 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff36370-d9ff-40ea-8fac-ae258781cc41 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:28.662349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff36370-d9ff-40ea-8fac-ae258781cc41 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:28.664380 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff36370-d9ff-40ea-8fac-ae258781cc41 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:28.673622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:28.678800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-bff36370-d9ff-40ea-8fac-ae258781cc41 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:28.678800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:28.678800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:28.678800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:28.678800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:28.678800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:28.678800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:28.678800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:28.678800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:28.678800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:28.678800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:28.678800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:28.678800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:28.678800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:28.678800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:28.678800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:28.680357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:28.680357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:28.680357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:28.680357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:28.680357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:28.680357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:28.680357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:28.680357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:28.680357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:28.680357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:28.680357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:28.680357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:28.680357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:28.680357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:28.680357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:28.680357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:28.680357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:28.681195 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:28.681195 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:28.681195 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:28.684690 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 248/497] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:28.685836 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf90026a-600d-4370-8169-3273fda44e2e None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:28.686118 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf90026a-600d-4370-8169-3273fda44e2e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:28.686414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf90026a-600d-4370-8169-3273fda44e2e None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:28.687670 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 249/498] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:28.688361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf90026a-600d-4370-8169-3273fda44e2e None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:28.721795 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb91046-61e8-4af8-8d24-4d893eb2cc73 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:28.723381 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:28.755949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fed4bc-5ea9-4e28-9822-19e199f658e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:28.756569 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fed4bc-5ea9-4e28-9822-19e199f658e6 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:28.756848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fed4bc-5ea9-4e28-9822-19e199f658e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:28.757235 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fed4bc-5ea9-4e28-9822-19e199f658e6 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:28.764820 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3fed4bc-5ea9-4e28-9822-19e199f658e6 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:28.766387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3fed4bc-5ea9-4e28-9822-19e199f658e6 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:28.779287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-c3fed4bc-5ea9-4e28-9822-19e199f658e6 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 23 13:43:28.779287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:28.779287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:28.779287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:28.779287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:28.779287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:28.779287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:28.779287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:28.779287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:28.779287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:28.779287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:28.779287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:28.779287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:43:28.779287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:43:28.779287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:43:28.779287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:43:28.780199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:28.780199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:28.780199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:43:28.780199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:28.780199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:43:28.780199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:28.780199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:28.780199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:28.780199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:28.780199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:28.780199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:28.780199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:28.780199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:28.780199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:28.780199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:28.780199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:28.780199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:28.780199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:28.781098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:28.781098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:28.781098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:43:28.781098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:43:28.781098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:43:28.781098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:43:28.781098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:43:28.781098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:43:28.781098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 23 13:43:28.781098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:28.781098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 250/499] 10.222.0.62 () {56 vars in 1093 bytes} [Fri Sep 23 13:43:28 2022] GET /identity/v3/users/nova => generated 82 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:28.782000 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b8c233-b42a-466c-a453-e7c55c52a54f None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:28.782000 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b8c233-b42a-466c-a453-e7c55c52a54f None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:28.782000 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b8c233-b42a-466c-a453-e7c55c52a54f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:28.795858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf90026a-600d-4370-8169-3273fda44e2e None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:28.800096 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-a9b8c233-b42a-466c-a453-e7c55c52a54f None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:28.801287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a9b8c233-b42a-466c-a453-e7c55c52a54f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:28.829566 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd8506d-10ba-4310-8f64-02730e5795da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:28.830168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd8506d-10ba-4310-8f64-02730e5795da None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:28.830473 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd8506d-10ba-4310-8f64-02730e5795da None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:28.830748 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd8506d-10ba-4310-8f64-02730e5795da None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:28.831383 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd8506d-10ba-4310-8f64-02730e5795da None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:28.831703 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd8506d-10ba-4310-8f64-02730e5795da None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:28.833068 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd8506d-10ba-4310-8f64-02730e5795da None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:28.844538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 250/500] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:28.845333 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd8506d-10ba-4310-8f64-02730e5795da None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:28.867404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 251/501] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:28.868662 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b7d38d-e3d1-4405-952c-f356906788f6 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:28.868953 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b7d38d-e3d1-4405-952c-f356906788f6 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:28.869223 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b7d38d-e3d1-4405-952c-f356906788f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:28.892563 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-05b7d38d-e3d1-4405-952c-f356906788f6 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:28.893753 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-05b7d38d-e3d1-4405-952c-f356906788f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:28.936394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:28.956520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:28.963010 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd8506d-10ba-4310-8f64-02730e5795da None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:28.974696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 252/502] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:28.975583 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b7d38d-e3d1-4405-952c-f356906788f6 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:29.002356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ea031e-9a15-4c92-b485-f39a888a01f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:29.003446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ea031e-9a15-4c92-b485-f39a888a01f0 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:29.003973 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ea031e-9a15-4c92-b485-f39a888a01f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:29.004443 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ea031e-9a15-4c92-b485-f39a888a01f0 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:29.005380 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ea031e-9a15-4c92-b485-f39a888a01f0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:29.005901 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ea031e-9a15-4c92-b485-f39a888a01f0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:29.008256 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ea031e-9a15-4c92-b485-f39a888a01f0 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:29.021233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 251/503] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:29.022278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ea031e-9a15-4c92-b485-f39a888a01f0 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:29.088842 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b7d38d-e3d1-4405-952c-f356906788f6 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:29.126207 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be7c036-5720-46e3-80aa-0fecf76ce173 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:29.126853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be7c036-5720-46e3-80aa-0fecf76ce173 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:29.127135 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be7c036-5720-46e3-80aa-0fecf76ce173 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:29.127453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be7c036-5720-46e3-80aa-0fecf76ce173 None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:29.128189 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be7c036-5720-46e3-80aa-0fecf76ce173 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:29.128495 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be7c036-5720-46e3-80aa-0fecf76ce173 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:29.129890 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ea031e-9a15-4c92-b485-f39a888a01f0 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:29.130362 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be7c036-5720-46e3-80aa-0fecf76ce173 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:29.167318 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a56416e-b46e-4b8f-a586-2fd103881ffb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:29.167942 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a56416e-b46e-4b8f-a586-2fd103881ffb None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:29.168222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a56416e-b46e-4b8f-a586-2fd103881ffb None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:29.168491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a56416e-b46e-4b8f-a586-2fd103881ffb None admin] PATH_INFO: `/v3/users/glance` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:29.171353 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 253/504] 10.222.0.62 () {56 vars in 1133 bytes} [Fri Sep 23 13:43:28 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:29.172254 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be7c036-5720-46e3-80aa-0fecf76ce173 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:29.180186 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a56416e-b46e-4b8f-a586-2fd103881ffb None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:29.181822 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a56416e-b46e-4b8f-a586-2fd103881ffb None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:29.196496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-4a56416e-b46e-4b8f-a586-2fd103881ffb None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 23 13:43:29.196496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:29.196496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:29.196496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:29.196496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:29.196496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:29.196496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:29.196496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:29.196496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:29.196496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:29.196496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:29.196496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:29.196496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:43:29.196496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:43:29.196496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:43:29.196496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:43:29.197409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:29.197409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:29.197409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:43:29.197409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:29.197409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:43:29.197409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:29.197409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:29.197409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:29.197409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:29.197409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:29.197409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:29.197409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:29.197409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:29.197409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:29.197409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:29.197409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:29.197409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:29.197409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:29.198305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:29.198305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:29.198305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:43:29.198305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:43:29.198305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:43:29.198305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:43:29.198305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:43:29.198305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:43:29.198305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 23 13:43:29.198305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:29.198305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 252/505] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:29 2022] GET /identity/v3/users/glance => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:29.198305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a56416e-b46e-4b8f-a586-2fd103881ffb None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:29.214879 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:29.221883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:29.289738 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be7c036-5720-46e3-80aa-0fecf76ce173 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:29.308564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a56416e-b46e-4b8f-a586-2fd103881ffb None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:29.323543 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363be0ac-6c17-4b2b-b72e-2e56441b72e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:29.324263 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363be0ac-6c17-4b2b-b72e-2e56441b72e6 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:29.324547 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363be0ac-6c17-4b2b-b72e-2e56441b72e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:29.324817 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363be0ac-6c17-4b2b-b72e-2e56441b72e6 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:29.331739 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363be0ac-6c17-4b2b-b72e-2e56441b72e6 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:29.333184 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363be0ac-6c17-4b2b-b72e-2e56441b72e6 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:29.342697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1d8413-bd2b-493f-8f2c-927b699295dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:29.343310 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1d8413-bd2b-493f-8f2c-927b699295dd None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:29.343582 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1d8413-bd2b-493f-8f2c-927b699295dd None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:29.343847 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1d8413-bd2b-493f-8f2c-927b699295dd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:29.346893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-363be0ac-6c17-4b2b-b72e-2e56441b72e6 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 23 13:43:29.346893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:29.346893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:29.346893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:29.346893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:29.346893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:29.346893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:29.346893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:29.346893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:29.346893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:29.346893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:29.346893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:29.346893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:43:29.346893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:43:29.346893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:43:29.346893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:43:29.347932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:29.347932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:29.347932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:43:29.347932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:29.347932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:43:29.347932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:29.347932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:29.347932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:29.347932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:29.347932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:29.347932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:29.347932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:29.347932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:29.347932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:29.347932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:29.347932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:29.347932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:29.347932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:29.348812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:29.348812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:29.348812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:43:29.348812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:43:29.348812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:43:29.348812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:43:29.348812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:43:29.348812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:43:29.348812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 23 13:43:29.348812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:29.348812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 254/506] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:29 2022] GET /identity/v3/users/cinder => generated 84 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:29.348812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363be0ac-6c17-4b2b-b72e-2e56441b72e6 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:29.356516 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1d8413-bd2b-493f-8f2c-927b699295dd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:29.361323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-9f1d8413-bd2b-493f-8f2c-927b699295dd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:29.361323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:29.361323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:29.361323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:29.361323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:29.361323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:29.361323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:29.361323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:29.361323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:29.361323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:29.361323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:29.361323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:29.361323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:29.361323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:29.361323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:29.361323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:29.362229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:29.362229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:29.362229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:29.362229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:29.362229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:29.362229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:29.362229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:29.362229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:29.362229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:29.362229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:29.362229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:29.362229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:29.362229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:29.362229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:29.362229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:29.362229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:29.363040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:29.363040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:29.363040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:29.363040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:29.363040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:29.363040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:29.363040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:29.363040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:29.363040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:29.363040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:29.363040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:29.363040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:29.363040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 253/507] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:43:29 2022] GET /identity/v3/roles/service => generated 85 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:29.363689 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1d8413-bd2b-493f-8f2c-927b699295dd None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:29.441234 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:29.442373 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:29.452375 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363be0ac-6c17-4b2b-b72e-2e56441b72e6 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:29.465154 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1d8413-bd2b-493f-8f2c-927b699295dd None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:29.482826 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b365723-2be2-4239-839b-c54f6d56fd44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:29.483426 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b365723-2be2-4239-839b-c54f6d56fd44 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:29.483704 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b365723-2be2-4239-839b-c54f6d56fd44 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:29.483970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b365723-2be2-4239-839b-c54f6d56fd44 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:29.500815 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b7868f-979b-461b-a1dd-06e13eb88f12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:29.501409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b7868f-979b-461b-a1dd-06e13eb88f12 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:29.501682 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b7868f-979b-461b-a1dd-06e13eb88f12 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:29.501973 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b7868f-979b-461b-a1dd-06e13eb88f12 None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:29.502744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b7868f-979b-461b-a1dd-06e13eb88f12 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:29.503045 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b7868f-979b-461b-a1dd-06e13eb88f12 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:29.504482 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b7868f-979b-461b-a1dd-06e13eb88f12 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:29.540822 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 254/508] 10.222.0.62 () {56 vars in 1137 bytes} [Fri Sep 23 13:43:29 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:29.541633 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b7868f-979b-461b-a1dd-06e13eb88f12 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:29.556305 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b365723-2be2-4239-839b-c54f6d56fd44 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:29.557744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b365723-2be2-4239-839b-c54f6d56fd44 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:29.643220 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b7868f-979b-461b-a1dd-06e13eb88f12 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:29.646944 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:29.656030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 255/509] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:29.660609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b365723-2be2-4239-839b-c54f6d56fd44 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:29.676647 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec83c55-77ad-4008-99d4-7a17cab481ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:29.681430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec83c55-77ad-4008-99d4-7a17cab481ff None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:29.681430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec83c55-77ad-4008-99d4-7a17cab481ff None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:29.681430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec83c55-77ad-4008-99d4-7a17cab481ff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:29.681430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:29.688999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec83c55-77ad-4008-99d4-7a17cab481ff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:29.690482 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec83c55-77ad-4008-99d4-7a17cab481ff None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:29.705221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-fec83c55-77ad-4008-99d4-7a17cab481ff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:29.705221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:29.705221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:29.705221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:29.705221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:29.705221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:29.705221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:29.705221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:29.705221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:29.705221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:29.705221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:29.705221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:29.705221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:29.705221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:29.705221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:29.705221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:29.706113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:29.706113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:29.706113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:29.706113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:29.706113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:29.706113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:29.706113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:29.706113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:29.706113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:29.706113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:29.706113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:29.706113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:29.706113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:29.706113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:29.706113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:29.706113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:29.706113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:29.707006 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:29.707006 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:29.707006 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:29.707006 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 255/510] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:29.707006 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec83c55-77ad-4008-99d4-7a17cab481ff None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:29.766112 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b365723-2be2-4239-839b-c54f6d56fd44 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:29.800344 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64159672-e039-4447-9406-0d0580a53556 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:29.800968 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64159672-e039-4447-9406-0d0580a53556 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:29.801246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64159672-e039-4447-9406-0d0580a53556 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:29.801516 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64159672-e039-4447-9406-0d0580a53556 None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:29.802243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64159672-e039-4447-9406-0d0580a53556 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:29.802575 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64159672-e039-4447-9406-0d0580a53556 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:29.804722 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64159672-e039-4447-9406-0d0580a53556 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:29.807785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec83c55-77ad-4008-99d4-7a17cab481ff None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:29.822297 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 256/511] 10.222.0.62 () {56 vars in 1137 bytes} [Fri Sep 23 13:43:29 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:29.822990 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64159672-e039-4447-9406-0d0580a53556 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:29.840265 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb19e7d-8fd3-40b4-8614-588900492f71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:29.840862 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb19e7d-8fd3-40b4-8614-588900492f71 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:29.841142 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb19e7d-8fd3-40b4-8614-588900492f71 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:29.841414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb19e7d-8fd3-40b4-8614-588900492f71 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:29.842074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb19e7d-8fd3-40b4-8614-588900492f71 None admin] RBAC: Adding query filter params (name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:29.842398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb19e7d-8fd3-40b4-8614-588900492f71 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:29.843776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb19e7d-8fd3-40b4-8614-588900492f71 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:29.852026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 256/512] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:43:29 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:29.852978 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb19e7d-8fd3-40b4-8614-588900492f71 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:29.879067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:29.901061 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:29.927566 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64159672-e039-4447-9406-0d0580a53556 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:29.950647 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb19e7d-8fd3-40b4-8614-588900492f71 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:29.961155 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6082dda2-88fb-4371-9d8f-e3579112f7d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:29.961696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6082dda2-88fb-4371-9d8f-e3579112f7d3 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:29.961975 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6082dda2-88fb-4371-9d8f-e3579112f7d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:29.962246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6082dda2-88fb-4371-9d8f-e3579112f7d3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:29.984917 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1b33ed-a338-4978-babb-7101e2c6f7e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:29.985542 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1b33ed-a338-4978-babb-7101e2c6f7e9 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:29.985816 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1b33ed-a338-4978-babb-7101e2c6f7e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:29.986085 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1b33ed-a338-4978-babb-7101e2c6f7e9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:29.997837 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1b33ed-a338-4978-babb-7101e2c6f7e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:29.999359 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1b33ed-a338-4978-babb-7101e2c6f7e9 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:30.013495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-ea1b33ed-a338-4978-babb-7101e2c6f7e9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:30.013495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:30.013495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:30.013495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:30.013495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:30.013495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:30.013495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:30.013495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:30.013495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:30.013495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:30.013495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:30.013495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:30.013495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:30.013495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:30.013495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:30.013495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:30.014524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:30.014524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:30.014524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:30.014524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:30.014524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:30.014524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:30.014524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:30.014524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:30.014524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:30.014524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:30.014524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:30.014524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:30.014524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:30.014524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:30.014524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:30.014524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:30.014524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:30.015369 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:30.015369 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:30.015369 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:30.015369 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 257/513] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:30.015961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1b33ed-a338-4978-babb-7101e2c6f7e9 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:30.035509 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6082dda2-88fb-4371-9d8f-e3579112f7d3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:30.036958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6082dda2-88fb-4371-9d8f-e3579112f7d3 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:30.085632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:30.113887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:30.118021 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1b33ed-a338-4978-babb-7101e2c6f7e9 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:30.136799 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 257/514] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:30.137639 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6082dda2-88fb-4371-9d8f-e3579112f7d3 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:30.153035 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11419343-70f9-422c-a6cc-71d9479d100a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:30.153633 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11419343-70f9-422c-a6cc-71d9479d100a None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:30.153907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11419343-70f9-422c-a6cc-71d9479d100a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:30.154172 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11419343-70f9-422c-a6cc-71d9479d100a None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:30.154850 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11419343-70f9-422c-a6cc-71d9479d100a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:30.155147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11419343-70f9-422c-a6cc-71d9479d100a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:30.156521 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11419343-70f9-422c-a6cc-71d9479d100a None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:30.168188 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 258/515] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:30.168969 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11419343-70f9-422c-a6cc-71d9479d100a None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:30.238113 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6082dda2-88fb-4371-9d8f-e3579112f7d3 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:30.268617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11419343-70f9-422c-a6cc-71d9479d100a None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:30.271719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75749814-609e-4e85-a166-9260f3e4ac49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:30.272316 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75749814-609e-4e85-a166-9260f3e4ac49 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:30.273088 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75749814-609e-4e85-a166-9260f3e4ac49 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:30.273347 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75749814-609e-4e85-a166-9260f3e4ac49 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:30.291743 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:30.302210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0f6c0b-1817-4e40-aaf8-caaa304c8c13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:30.302857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0f6c0b-1817-4e40-aaf8-caaa304c8c13 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:30.303132 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0f6c0b-1817-4e40-aaf8-caaa304c8c13 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:30.307553 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0f6c0b-1817-4e40-aaf8-caaa304c8c13 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:30.313678 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0f6c0b-1817-4e40-aaf8-caaa304c8c13 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:30.315187 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0f6c0b-1817-4e40-aaf8-caaa304c8c13 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:30.328885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-7c0f6c0b-1817-4e40-aaf8-caaa304c8c13 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:30.328885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:30.328885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:30.328885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:30.328885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:30.328885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:30.328885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:30.328885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:30.328885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:30.328885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:30.328885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:30.328885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:30.328885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:30.328885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:30.328885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:30.328885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:30.329927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:30.329927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:30.329927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:30.329927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:30.329927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:30.329927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:30.329927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:30.329927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:30.329927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:30.329927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:30.329927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:30.329927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:30.329927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:30.329927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:30.329927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:30.329927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:30.329927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:30.330987 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:30.330987 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:30.330987 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:30.330987 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 259/516] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:30.330987 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0f6c0b-1817-4e40-aaf8-caaa304c8c13 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:30.342455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75749814-609e-4e85-a166-9260f3e4ac49 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:30.344222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75749814-609e-4e85-a166-9260f3e4ac49 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:30.345907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:30.434800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0f6c0b-1817-4e40-aaf8-caaa304c8c13 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:30.440911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 258/517] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:30.441573 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75749814-609e-4e85-a166-9260f3e4ac49 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:30.469707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7efeba-f841-47e9-b2fc-a7215d6243ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:30.470233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7efeba-f841-47e9-b2fc-a7215d6243ca None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:30.470538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7efeba-f841-47e9-b2fc-a7215d6243ca None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:30.470809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7efeba-f841-47e9-b2fc-a7215d6243ca None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:30.471443 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7efeba-f841-47e9-b2fc-a7215d6243ca None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:30.471737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7efeba-f841-47e9-b2fc-a7215d6243ca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:30.473088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7efeba-f841-47e9-b2fc-a7215d6243ca None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:30.484097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 260/518] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:30.484966 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7efeba-f841-47e9-b2fc-a7215d6243ca None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:30.494175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:30.543102 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75749814-609e-4e85-a166-9260f3e4ac49 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:30.555473 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:30.576216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f40bab-1975-4e72-ba79-ca14c0175134 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:30.576841 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f40bab-1975-4e72-ba79-ca14c0175134 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:30.577119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f40bab-1975-4e72-ba79-ca14c0175134 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:30.577393 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f40bab-1975-4e72-ba79-ca14c0175134 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:30.583793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7efeba-f841-47e9-b2fc-a7215d6243ca None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:30.587266 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f40bab-1975-4e72-ba79-ca14c0175134 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:30.588740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f40bab-1975-4e72-ba79-ca14c0175134 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:30.604139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-c2f40bab-1975-4e72-ba79-ca14c0175134 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:30.604139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:30.604139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:30.604139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:30.604139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:30.604139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:30.604139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:30.604139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:30.604139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:30.604139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:30.604139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:30.604139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:30.604139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:30.604139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:30.604139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:30.604139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:30.605153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:30.605153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:30.605153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:30.605153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:30.605153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:30.605153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:30.605153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:30.605153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:30.605153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:30.605153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:30.605153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:30.605153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:30.605153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:30.605153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:30.605153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:30.605153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:30.605153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:30.606165 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:30.606165 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:30.606165 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:30.606165 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 259/519] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:30.606448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f40bab-1975-4e72-ba79-ca14c0175134 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:30.623320 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d06d026-5159-4e99-beeb-ab804144378a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:30.623916 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d06d026-5159-4e99-beeb-ab804144378a None admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:30.624191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d06d026-5159-4e99-beeb-ab804144378a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:30.624456 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d06d026-5159-4e99-beeb-ab804144378a None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:30.625093 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d06d026-5159-4e99-beeb-ab804144378a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:30.626526 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d06d026-5159-4e99-beeb-ab804144378a None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:30.702852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f40bab-1975-4e72-ba79-ca14c0175134 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:30.704337 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:30.734238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 261/520] 10.222.0.62 () {60 vars in 1132 bytes} [Fri Sep 23 13:43:30 2022] POST /identity/v3/users => generated 247 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:43:30.734825 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d06d026-5159-4e99-beeb-ab804144378a None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:30.735713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884bf696-e403-476b-8119-5d99286f36cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:30.736313 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884bf696-e403-476b-8119-5d99286f36cb None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:30.736589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884bf696-e403-476b-8119-5d99286f36cb None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:30.736867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884bf696-e403-476b-8119-5d99286f36cb None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:30.737500 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884bf696-e403-476b-8119-5d99286f36cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:30.737797 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884bf696-e403-476b-8119-5d99286f36cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:30.739234 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884bf696-e403-476b-8119-5d99286f36cb None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:30.759050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 260/521] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:30.759050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884bf696-e403-476b-8119-5d99286f36cb None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:30.869554 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884bf696-e403-476b-8119-5d99286f36cb None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:30.884301 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d06d026-5159-4e99-beeb-ab804144378a None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:30.909996 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d537483f-f671-4fe0-9b6a-323689a71148 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:30.910899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d537483f-f671-4fe0-9b6a-323689a71148 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:30.911453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d537483f-f671-4fe0-9b6a-323689a71148 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:30.911453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d537483f-f671-4fe0-9b6a-323689a71148 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:30.915869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:30.918961 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d537483f-f671-4fe0-9b6a-323689a71148 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:30.920397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d537483f-f671-4fe0-9b6a-323689a71148 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:30.924488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371a39f8-ccdc-4336-9f7d-b941c489d803 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:30.925116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371a39f8-ccdc-4336-9f7d-b941c489d803 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:30.925391 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371a39f8-ccdc-4336-9f7d-b941c489d803 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:30.925669 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371a39f8-ccdc-4336-9f7d-b941c489d803 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:30.932902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-d537483f-f671-4fe0-9b6a-323689a71148 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:30.932902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:30.932902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:30.932902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:30.932902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:30.932902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:30.932902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:30.932902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:30.932902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:30.932902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:30.932902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:30.932902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:30.932902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:43:30.932902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:43:30.932902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:43:30.934179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:43:30.934179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:30.934179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:30.934179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:30.934179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:30.934179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:30.934179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:30.934179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:30.934179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:30.934179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:30.934179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:30.934179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:30.934179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:30.934179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:30.934179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:30.934179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:30.934179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:30.935070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:43:30.935070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:43:30.935070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:43:30.935070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:43:30.935070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:43:30.935070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:43:30.935070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:30.935070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:30.935070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 261/522] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:30.935559 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d537483f-f671-4fe0-9b6a-323689a71148 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:30.941326 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371a39f8-ccdc-4336-9f7d-b941c489d803 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:30.942921 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371a39f8-ccdc-4336-9f7d-b941c489d803 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:30.958828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-371a39f8-ccdc-4336-9f7d-b941c489d803 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:30.958828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:30.958828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:30.958828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:30.958828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:30.958828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:30.958828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:30.958828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:30.958828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:30.958828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:30.958828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:30.958828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:30.958828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:30.958828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:30.958828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:30.958828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:30.960477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:30.960477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:30.960477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:30.960477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:30.960477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:30.960477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:30.960477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:30.960477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:30.960477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:30.960477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:30.960477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:30.960477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:30.960477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:30.960477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:30.960477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:30.960477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:30.960477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:30.961607 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:30.961607 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:30.961607 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:30.961607 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 262/523] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:30.961607 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371a39f8-ccdc-4336-9f7d-b941c489d803 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:30.968497 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:31.059839 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d537483f-f671-4fe0-9b6a-323689a71148 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:31.071376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371a39f8-ccdc-4336-9f7d-b941c489d803 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:31.098287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabee0f0-1711-4cc3-b77f-1b7b48013781 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:31.098929 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabee0f0-1711-4cc3-b77f-1b7b48013781 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:31.099202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabee0f0-1711-4cc3-b77f-1b7b48013781 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:31.099464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabee0f0-1711-4cc3-b77f-1b7b48013781 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:31.100216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fabee0f0-1711-4cc3-b77f-1b7b48013781 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:31.100644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fabee0f0-1711-4cc3-b77f-1b7b48013781 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:31.102246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fabee0f0-1711-4cc3-b77f-1b7b48013781 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:31.105333 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087a07a1-264c-451d-aedb-f9443b6890f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:31.105954 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087a07a1-264c-451d-aedb-f9443b6890f6 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:31.106227 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087a07a1-264c-451d-aedb-f9443b6890f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:31.106523 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-087a07a1-264c-451d-aedb-f9443b6890f6 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:31.114253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087a07a1-264c-451d-aedb-f9443b6890f6 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:31.114906 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 262/524] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:31.115706 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabee0f0-1711-4cc3-b77f-1b7b48013781 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:31.115805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-087a07a1-264c-451d-aedb-f9443b6890f6 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:31.130364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-087a07a1-264c-451d-aedb-f9443b6890f6 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 23 13:43:31.130364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:31.130364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:31.130364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:31.130364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:31.130364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:31.130364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:31.130364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:31.130364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:31.130364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:31.130364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:31.130364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:31.130364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:43:31.130364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:43:31.130364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:43:31.130364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:43:31.131281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:31.131281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:31.131281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:43:31.131281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:31.131281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:43:31.131281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:31.131281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:31.131281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:31.131281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:31.131281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:31.131281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:31.131281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:31.131281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:31.131281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:31.131281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:31.131281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:31.131281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:31.131281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:31.132792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:31.132792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:31.132792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:43:31.132792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:43:31.132792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:43:31.132792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:43:31.132792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:43:31.132792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:43:31.132792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 23 13:43:31.132792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:31.132792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 263/525] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:43:30 2022] GET /identity/v3/users/neutron => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:31.132792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087a07a1-264c-451d-aedb-f9443b6890f6 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:31.176399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:31.214162 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:31.233632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabee0f0-1711-4cc3-b77f-1b7b48013781 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:31.244869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-087a07a1-264c-451d-aedb-f9443b6890f6 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:31.274366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f89aec-b20b-4265-9497-31d6c83e7d27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:31.275150 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f89aec-b20b-4265-9497-31d6c83e7d27 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:31.275540 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f89aec-b20b-4265-9497-31d6c83e7d27 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:31.275905 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f89aec-b20b-4265-9497-31d6c83e7d27 None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:31.276672 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f89aec-b20b-4265-9497-31d6c83e7d27 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:31.277044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f89aec-b20b-4265-9497-31d6c83e7d27 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:31.278580 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f89aec-b20b-4265-9497-31d6c83e7d27 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:31.283957 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b4460f-85b4-41e5-8490-89c831171a6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:31.284556 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b4460f-85b4-41e5-8490-89c831171a6d None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:31.284833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b4460f-85b4-41e5-8490-89c831171a6d None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:31.285102 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b4460f-85b4-41e5-8490-89c831171a6d None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:31.285739 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b4460f-85b4-41e5-8490-89c831171a6d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:31.286040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b4460f-85b4-41e5-8490-89c831171a6d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:31.287417 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b4460f-85b4-41e5-8490-89c831171a6d None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:31.298247 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 263/526] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:43:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:31.298930 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f89aec-b20b-4265-9497-31d6c83e7d27 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:31.305555 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 264/527] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:31.306675 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b4460f-85b4-41e5-8490-89c831171a6d None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:31.408786 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f89aec-b20b-4265-9497-31d6c83e7d27 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:31.410321 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:31.414640 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b4460f-85b4-41e5-8490-89c831171a6d None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:31.444081 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98024ab1-c5a0-4922-ba6b-65b0f9c83841 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:31.445338 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98024ab1-c5a0-4922-ba6b-65b0f9c83841 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:31.445702 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98024ab1-c5a0-4922-ba6b-65b0f9c83841 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:31.446066 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98024ab1-c5a0-4922-ba6b-65b0f9c83841 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:31.451300 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6400fc97-cc17-436e-ae62-f2b8d9c87872 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:31.451970 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6400fc97-cc17-436e-ae62-f2b8d9c87872 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:31.452319 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6400fc97-cc17-436e-ae62-f2b8d9c87872 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:31.452649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6400fc97-cc17-436e-ae62-f2b8d9c87872 None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:31.453053 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98024ab1-c5a0-4922-ba6b-65b0f9c83841 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:31.453501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6400fc97-cc17-436e-ae62-f2b8d9c87872 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:31.453866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6400fc97-cc17-436e-ae62-f2b8d9c87872 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:31.454698 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98024ab1-c5a0-4922-ba6b-65b0f9c83841 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:31.455424 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6400fc97-cc17-436e-ae62-f2b8d9c87872 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:31.468868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-98024ab1-c5a0-4922-ba6b-65b0f9c83841 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:31.468868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:31.468868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:31.468868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:31.468868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:31.468868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:31.468868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:31.468868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:31.468868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:31.468868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:31.468868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:31.468868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:31.468868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:43:31.468868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:43:31.468868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:43:31.469889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:43:31.469889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:31.469889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:31.469889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:31.469889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:31.469889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:31.469889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:31.469889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:31.469889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:31.469889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:31.469889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:31.469889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:31.469889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:31.469889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:31.469889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:31.469889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:31.469889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:31.470969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:43:31.470969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:43:31.470969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:43:31.470969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:43:31.470969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:43:31.470969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:43:31.470969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:31.470969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:31.480278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 265/528] 10.222.0.62 () {56 vars in 1139 bytes} [Fri Sep 23 13:43:31 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:31.480441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 264/529] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:31.480525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6400fc97-cc17-436e-ae62-f2b8d9c87872 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:31.480595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98024ab1-c5a0-4922-ba6b-65b0f9c83841 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:31.480595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:31.585832 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98024ab1-c5a0-4922-ba6b-65b0f9c83841 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:31.586882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6400fc97-cc17-436e-ae62-f2b8d9c87872 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:31.622106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b24162e-a0d7-48fe-8ca6-8bf173fb0d0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:31.622739 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b24162e-a0d7-48fe-8ca6-8bf173fb0d0d None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:31.623013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b24162e-a0d7-48fe-8ca6-8bf173fb0d0d None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:31.623278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b24162e-a0d7-48fe-8ca6-8bf173fb0d0d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:31.623996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b24162e-a0d7-48fe-8ca6-8bf173fb0d0d None admin] RBAC: Adding query filter params (role.id=809612fc6d62442cbcc179b5c4123465, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=f01ed94dbb384a178ad0a45a6f3d8c17) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:31.624296 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b24162e-a0d7-48fe-8ca6-8bf173fb0d0d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:31.624667 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b555c445-94ac-4840-94ab-a01a8a2ceb25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:31.625462 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b555c445-94ac-4840-94ab-a01a8a2ceb25 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:31.625713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b24162e-a0d7-48fe-8ca6-8bf173fb0d0d None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:31.626046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b555c445-94ac-4840-94ab-a01a8a2ceb25 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:31.626488 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b555c445-94ac-4840-94ab-a01a8a2ceb25 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:31.631048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 266/530] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:43:31 2022] GET /identity/v3/role_assignments?user.id=f01ed94dbb384a178ad0a45a6f3d8c17&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=809612fc6d62442cbcc179b5c4123465 => generated 259 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:31.631855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b24162e-a0d7-48fe-8ca6-8bf173fb0d0d None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:31.640397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b555c445-94ac-4840-94ab-a01a8a2ceb25 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:31.641989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b555c445-94ac-4840-94ab-a01a8a2ceb25 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:31.655595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-b555c445-94ac-4840-94ab-a01a8a2ceb25 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:31.655595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:31.655595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:31.655595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:31.655595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:31.655595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:31.655595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:31.655595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:31.655595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:31.655595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:31.655595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:31.655595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:31.655595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:43:31.655595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:43:31.655595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:43:31.656502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:43:31.656502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:31.656502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:31.656502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:31.656502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:31.656502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:31.656502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:31.656502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:31.656502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:31.656502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:31.656502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:31.656502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:31.656502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:31.656502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:31.656502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:31.656502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:31.656502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:31.657332 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:43:31.657332 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:43:31.657332 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:43:31.657332 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:43:31.657332 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:43:31.657332 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:43:31.657332 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:31.657332 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:31.659634 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 265/531] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:31.660817 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b555c445-94ac-4840-94ab-a01a8a2ceb25 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:31.672393 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:31.735918 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:31.767672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b24162e-a0d7-48fe-8ca6-8bf173fb0d0d None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:31.782255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b555c445-94ac-4840-94ab-a01a8a2ceb25 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:31.818580 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de58c0d7-8a7e-44b3-9442-c7058b5da5f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:31.819628 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de58c0d7-8a7e-44b3-9442-c7058b5da5f0 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:31.820079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de58c0d7-8a7e-44b3-9442-c7058b5da5f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:31.820700 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de58c0d7-8a7e-44b3-9442-c7058b5da5f0 None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:31.821265 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0469a5-0d96-44eb-9e97-55ecfcc73361 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:31.821804 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de58c0d7-8a7e-44b3-9442-c7058b5da5f0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:31.821976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0469a5-0d96-44eb-9e97-55ecfcc73361 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:31.822423 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0469a5-0d96-44eb-9e97-55ecfcc73361 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:31.822651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de58c0d7-8a7e-44b3-9442-c7058b5da5f0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:31.822748 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0469a5-0d96-44eb-9e97-55ecfcc73361 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:31.824367 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de58c0d7-8a7e-44b3-9442-c7058b5da5f0 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:31.851548 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 267/532] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:43:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:31.852507 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de58c0d7-8a7e-44b3-9442-c7058b5da5f0 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:31.908027 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0469a5-0d96-44eb-9e97-55ecfcc73361 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:31.909511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0469a5-0d96-44eb-9e97-55ecfcc73361 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:31.971970 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:31.975043 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:31.998699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de58c0d7-8a7e-44b3-9442-c7058b5da5f0 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:32.016371 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 266/533] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:32.017219 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0469a5-0d96-44eb-9e97-55ecfcc73361 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:32.046589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb73d3eb-ac7c-43f1-b863-be4b97ad3451 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:32.047584 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb73d3eb-ac7c-43f1-b863-be4b97ad3451 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:32.048046 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb73d3eb-ac7c-43f1-b863-be4b97ad3451 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:32.048745 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb73d3eb-ac7c-43f1-b863-be4b97ad3451 None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:32.049861 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb73d3eb-ac7c-43f1-b863-be4b97ad3451 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:32.050322 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb73d3eb-ac7c-43f1-b863-be4b97ad3451 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:32.052879 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb73d3eb-ac7c-43f1-b863-be4b97ad3451 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:32.073674 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 268/534] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:43:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:32.074291 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb73d3eb-ac7c-43f1-b863-be4b97ad3451 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:32.128216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0469a5-0d96-44eb-9e97-55ecfcc73361 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:32.173389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b97ed86-ac12-41fb-bdcc-71000daccf02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:32.174888 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b97ed86-ac12-41fb-bdcc-71000daccf02 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:32.175418 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b97ed86-ac12-41fb-bdcc-71000daccf02 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:32.176340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b97ed86-ac12-41fb-bdcc-71000daccf02 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:32.177681 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b97ed86-ac12-41fb-bdcc-71000daccf02 None admin] RBAC: Adding query filter params (role.id=809612fc6d62442cbcc179b5c4123465, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=acdf4a7bd608492390176dcc42ca1267) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:32.178352 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b97ed86-ac12-41fb-bdcc-71000daccf02 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:32.180679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b97ed86-ac12-41fb-bdcc-71000daccf02 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:32.194956 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 267/535] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:43:32 2022] GET /identity/v3/role_assignments?user.id=acdf4a7bd608492390176dcc42ca1267&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=809612fc6d62442cbcc179b5c4123465 => generated 259 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:32.194956 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b97ed86-ac12-41fb-bdcc-71000daccf02 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:32.224997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb73d3eb-ac7c-43f1-b863-be4b97ad3451 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:32.255459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:32.337031 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73b391c-88a1-4df2-ab89-c03f94cdfffa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:32.338078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73b391c-88a1-4df2-ab89-c03f94cdfffa None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:32.338589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73b391c-88a1-4df2-ab89-c03f94cdfffa None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:32.339067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73b391c-88a1-4df2-ab89-c03f94cdfffa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:32.345256 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b97ed86-ac12-41fb-bdcc-71000daccf02 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:32.362519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73b391c-88a1-4df2-ab89-c03f94cdfffa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:32.364312 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73b391c-88a1-4df2-ab89-c03f94cdfffa None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:32.372965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:32.398429 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe182ad-446a-4768-9c58-7847587dde63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:32.399069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe182ad-446a-4768-9c58-7847587dde63 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:32.399348 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe182ad-446a-4768-9c58-7847587dde63 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:32.399620 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe182ad-446a-4768-9c58-7847587dde63 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:32.400370 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe182ad-446a-4768-9c58-7847587dde63 None admin] RBAC: Adding query filter params (role.id=809612fc6d62442cbcc179b5c4123465, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=fae68ca0754a4b0b88a57f39efa2fb45) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:32.400683 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe182ad-446a-4768-9c58-7847587dde63 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:32.402142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe182ad-446a-4768-9c58-7847587dde63 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:32.405735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-c73b391c-88a1-4df2-ab89-c03f94cdfffa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:32.405735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:32.405735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:32.405735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:32.405735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:32.405735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:32.405735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:32.405735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:32.405735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:32.405735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:32.405735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:32.405735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:32.405735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:32.405735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:32.405735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:32.405735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:32.406657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:32.406657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:32.406657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:32.406657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:32.406657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:32.406657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:32.406657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:32.406657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:32.406657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:32.406657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:32.406657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:32.406657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:32.406657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:32.406657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:32.406657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:32.406657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:32.406657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:32.407514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:32.407514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:32.407514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:32.407514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 269/536] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 333 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:32.413752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73b391c-88a1-4df2-ab89-c03f94cdfffa None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:32.419664 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 268/537] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:43:32 2022] GET /identity/v3/role_assignments?user.id=fae68ca0754a4b0b88a57f39efa2fb45&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=809612fc6d62442cbcc179b5c4123465 => generated 259 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:32.558291 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73b391c-88a1-4df2-ab89-c03f94cdfffa None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:32.611238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8878e11-dd80-4d8d-bdb8-7458ae82b468 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:32.611779 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8878e11-dd80-4d8d-bdb8-7458ae82b468 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:32.612057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8878e11-dd80-4d8d-bdb8-7458ae82b468 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:32.612327 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8878e11-dd80-4d8d-bdb8-7458ae82b468 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:32.612986 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8878e11-dd80-4d8d-bdb8-7458ae82b468 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:32.613286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8878e11-dd80-4d8d-bdb8-7458ae82b468 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:32.614705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8878e11-dd80-4d8d-bdb8-7458ae82b468 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:32.628537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 270/538] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:32.634731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe182ad-446a-4768-9c58-7847587dde63 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:32.769435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:32.781403 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe182ad-446a-4768-9c58-7847587dde63 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:32.832283 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7e6f2e-77ef-40e3-8153-24ba9404c14a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:32.832841 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7e6f2e-77ef-40e3-8153-24ba9404c14a None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:32.833119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7e6f2e-77ef-40e3-8153-24ba9404c14a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:32.833387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7e6f2e-77ef-40e3-8153-24ba9404c14a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:32.843983 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7e6f2e-77ef-40e3-8153-24ba9404c14a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:32.845425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7e6f2e-77ef-40e3-8153-24ba9404c14a None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:32.861420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-8a7e6f2e-77ef-40e3-8153-24ba9404c14a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:32.861420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:32.861420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:32.861420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:32.861420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:32.861420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:32.861420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:32.861420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:32.861420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:32.861420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:32.861420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:32.861420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:32.861420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:43:32.861420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:43:32.861420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:43:32.862323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:43:32.862323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:32.862323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:32.862323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:32.862323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:32.862323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:32.862323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:32.862323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:32.862323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:32.862323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:32.862323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:32.862323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:32.862323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:32.862323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:32.862323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:32.862323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:32.862323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:32.863244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:43:32.863244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:43:32.863244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:43:32.863244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:43:32.863244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:43:32.863244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:43:32.863244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:32.863244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:32.863244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 269/539] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:32.869348 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8878e11-dd80-4d8d-bdb8-7458ae82b468 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:32.927917 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:32.990793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8878e11-dd80-4d8d-bdb8-7458ae82b468 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:33.042737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38408ff-a89b-45a5-a958-fd9e59e6aa77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:33.043702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38408ff-a89b-45a5-a958-fd9e59e6aa77 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:33.044183 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38408ff-a89b-45a5-a958-fd9e59e6aa77 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:33.044676 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38408ff-a89b-45a5-a958-fd9e59e6aa77 None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:33.045578 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38408ff-a89b-45a5-a958-fd9e59e6aa77 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:33.046087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38408ff-a89b-45a5-a958-fd9e59e6aa77 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:33.047909 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38408ff-a89b-45a5-a958-fd9e59e6aa77 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:33.073497 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 271/540] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:43:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:33.079682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7e6f2e-77ef-40e3-8153-24ba9404c14a None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:33.179926 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724473fe-7906-4d95-878a-56277d72e7a9 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:33.180534 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724473fe-7906-4d95-878a-56277d72e7a9 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:33.181023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724473fe-7906-4d95-878a-56277d72e7a9 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:33.183260 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 272/541] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:33 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:33.189038 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ff17ee-ec8a-4b54-9fe4-e686de634029 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:33.189484 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ff17ee-ec8a-4b54-9fe4-e686de634029 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:33.189756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ff17ee-ec8a-4b54-9fe4-e686de634029 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:33.223431 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c9ff17ee-ec8a-4b54-9fe4-e686de634029 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:33.224663 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c9ff17ee-ec8a-4b54-9fe4-e686de634029 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:33.244922 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7e6f2e-77ef-40e3-8153-24ba9404c14a None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:33.287766 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28f5365-29cd-4a94-af46-dfa96b2ccd77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:33.288930 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28f5365-29cd-4a94-af46-dfa96b2ccd77 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:33.289535 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28f5365-29cd-4a94-af46-dfa96b2ccd77 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:33.290007 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28f5365-29cd-4a94-af46-dfa96b2ccd77 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:33.290979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28f5365-29cd-4a94-af46-dfa96b2ccd77 None admin] RBAC: Adding query filter params (role.id=809612fc6d62442cbcc179b5c4123465, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=580a35627c234a25aa04f5d376177ca5) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:33.291504 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28f5365-29cd-4a94-af46-dfa96b2ccd77 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:33.293119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28f5365-29cd-4a94-af46-dfa96b2ccd77 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:33.302661 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 270/542] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:43:33 2022] GET /identity/v3/role_assignments?user.id=580a35627c234a25aa04f5d376177ca5&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=809612fc6d62442cbcc179b5c4123465 => generated 259 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:33.348670 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 273/543] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:33.640803 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28f5365-29cd-4a94-af46-dfa96b2ccd77 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:33.660007 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:33.752566 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28f5365-29cd-4a94-af46-dfa96b2ccd77 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:33.790543 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04db599f-2141-4931-9a7e-8eafa8ce7139 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:33.791527 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04db599f-2141-4931-9a7e-8eafa8ce7139 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:33.792034 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04db599f-2141-4931-9a7e-8eafa8ce7139 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:33.792524 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04db599f-2141-4931-9a7e-8eafa8ce7139 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:33.803312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04db599f-2141-4931-9a7e-8eafa8ce7139 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:33.809400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-04db599f-2141-4931-9a7e-8eafa8ce7139 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:33.809400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:33.809400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:33.809400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:33.809400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:33.809400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:33.809400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:33.809400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:33.809400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:33.809400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:33.809400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:33.809400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:33.809400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:33.809400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:33.809400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:33.809400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:33.810273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:33.810273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:33.810273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:33.810273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:33.810273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:33.810273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:33.810273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:33.810273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:33.810273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:33.810273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:33.810273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:33.810273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:33.810273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:33.810273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:33.810273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:33.810273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:33.811087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:33.811087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:33.811087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:33.811087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:33.811087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:33.811087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:33.811087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:33.811087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:33.811087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:33.811087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:33.811087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:33.811087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:33.813601 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 271/544] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:43:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:33.820755 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ff17ee-ec8a-4b54-9fe4-e686de634029 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:33.869852 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:33.928989 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ff17ee-ec8a-4b54-9fe4-e686de634029 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:33.980117 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d450b0-f939-4035-a8a6-5693ad06b0c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:33.981137 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d450b0-f939-4035-a8a6-5693ad06b0c9 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:33.981619 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d450b0-f939-4035-a8a6-5693ad06b0c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:33.982101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d450b0-f939-4035-a8a6-5693ad06b0c9 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:33.983020 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d450b0-f939-4035-a8a6-5693ad06b0c9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:33.983594 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d450b0-f939-4035-a8a6-5693ad06b0c9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:33.985184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d450b0-f939-4035-a8a6-5693ad06b0c9 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:33.998645 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 274/545] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:43:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:34.004681 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04db599f-2141-4931-9a7e-8eafa8ce7139 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:34.102697 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:34.106114 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a9cd8a-3a89-4821-86cb-808d86cd91a6 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:34.106641 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a9cd8a-3a89-4821-86cb-808d86cd91a6 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:34.107142 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a9cd8a-3a89-4821-86cb-808d86cd91a6 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:34.109461 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 275/546] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:34.116895 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3036f43-f2d5-4956-b145-e963eef377af None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:34.117570 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3036f43-f2d5-4956-b145-e963eef377af None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:34.118367 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3036f43-f2d5-4956-b145-e963eef377af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:34.132083 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04db599f-2141-4931-9a7e-8eafa8ce7139 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:34.170940 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46da8f4-4fbf-4faa-b830-36966af89b32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:34.171626 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46da8f4-4fbf-4faa-b830-36966af89b32 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:34.171907 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46da8f4-4fbf-4faa-b830-36966af89b32 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:34.172177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46da8f4-4fbf-4faa-b830-36966af89b32 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:34.182769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46da8f4-4fbf-4faa-b830-36966af89b32 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:34.184229 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46da8f4-4fbf-4faa-b830-36966af89b32 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:34.196706 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c3036f43-f2d5-4956-b145-e963eef377af None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:34.197909 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c3036f43-f2d5-4956-b145-e963eef377af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:34.233640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-c46da8f4-4fbf-4faa-b830-36966af89b32 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:34.233640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:34.233640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:34.233640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:34.233640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:34.233640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:34.233640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:34.233640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:34.233640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:34.233640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:34.233640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:34.233640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:34.233640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:34.233640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:34.233640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:34.233640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:34.234623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:34.234623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:34.234623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:34.234623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:34.234623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:34.234623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:34.234623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:34.234623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:34.234623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:34.234623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:34.234623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:34.234623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:34.234623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:34.234623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:34.234623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:34.234623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:34.234623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:34.235865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:34.235865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:34.235865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:34.243754 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 272/547] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:34.244965 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46da8f4-4fbf-4faa-b830-36966af89b32 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:34.339438 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 276/548] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:34.390016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46da8f4-4fbf-4faa-b830-36966af89b32 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:34.417323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:34.426968 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0054f116-dc86-4459-aee0-407f312113d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:34.427881 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0054f116-dc86-4459-aee0-407f312113d6 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:34.428543 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0054f116-dc86-4459-aee0-407f312113d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:34.429500 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0054f116-dc86-4459-aee0-407f312113d6 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:34.430310 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0054f116-dc86-4459-aee0-407f312113d6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:34.433890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0054f116-dc86-4459-aee0-407f312113d6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:34.433890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0054f116-dc86-4459-aee0-407f312113d6 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:34.450181 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 273/549] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:34.452042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3036f43-f2d5-4956-b145-e963eef377af None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:34.517621 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:34.607128 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3036f43-f2d5-4956-b145-e963eef377af None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:34.660064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e712b681-93fc-4ace-b0e9-d4989cec1fc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:34.661083 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e712b681-93fc-4ace-b0e9-d4989cec1fc1 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:34.661571 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e712b681-93fc-4ace-b0e9-d4989cec1fc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:34.662032 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e712b681-93fc-4ace-b0e9-d4989cec1fc1 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:34.675829 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e712b681-93fc-4ace-b0e9-d4989cec1fc1 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:34.677680 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e712b681-93fc-4ace-b0e9-d4989cec1fc1 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:34.680896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3cea99-40fe-428a-bb32-01798aa55770 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:34.681475 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3cea99-40fe-428a-bb32-01798aa55770 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:34.681947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3cea99-40fe-428a-bb32-01798aa55770 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:34.683830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 274/550] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:43:34.691265 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b048795a-90df-4db3-aa03-53433e7e2509 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:34.691809 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b048795a-90df-4db3-aa03-53433e7e2509 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:34.692296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b048795a-90df-4db3-aa03-53433e7e2509 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:34.698351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-e712b681-93fc-4ace-b0e9-d4989cec1fc1 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 23 13:43:34.698351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:34.698351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:34.698351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:34.698351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:34.698351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:34.698351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:34.698351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:34.698351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:34.698351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:34.698351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:34.698351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:34.698351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:43:34.698351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:43:34.698351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:43:34.698351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:43:34.699251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:34.699251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:34.699251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:43:34.699251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:34.699251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:43:34.699251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:34.699251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:34.699251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:34.699251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:34.699251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:34.699251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:34.699251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:34.699251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:34.699251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:34.699251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:34.699251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:34.699251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:34.699251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:34.700144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:34.700144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:34.700144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:43:34.700144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:43:34.700144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:43:34.700144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:43:34.700144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:43:34.700144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:43:34.700144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 23 13:43:34.700144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:34.701262 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 277/551] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:43:34 2022] GET /identity/v3/users/swift => generated 83 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:34.703102 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e712b681-93fc-4ace-b0e9-d4989cec1fc1 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:34.718516 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-b048795a-90df-4db3-aa03-53433e7e2509 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:34.720005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-b048795a-90df-4db3-aa03-53433e7e2509 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:34.809173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 275/552] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:34.810799 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b048795a-90df-4db3-aa03-53433e7e2509 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:34.831806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:34.841065 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e712b681-93fc-4ace-b0e9-d4989cec1fc1 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:34.887863 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f659e18-c49f-4ec9-998a-817371408e87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:34.888904 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f659e18-c49f-4ec9-998a-817371408e87 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:34.889255 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f659e18-c49f-4ec9-998a-817371408e87 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:34.889592 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f659e18-c49f-4ec9-998a-817371408e87 None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:34.890425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f659e18-c49f-4ec9-998a-817371408e87 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:34.890932 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f659e18-c49f-4ec9-998a-817371408e87 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:34.893215 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f659e18-c49f-4ec9-998a-817371408e87 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:34.915624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 278/553] 10.222.0.62 () {56 vars in 1135 bytes} [Fri Sep 23 13:43:34 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:34.925662 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f659e18-c49f-4ec9-998a-817371408e87 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:34.931008 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b048795a-90df-4db3-aa03-53433e7e2509 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:34.950758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:34.971048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c058bc-455d-422c-bf53-f468012c4889 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:34.977308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c058bc-455d-422c-bf53-f468012c4889 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:34.977308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c058bc-455d-422c-bf53-f468012c4889 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:34.977308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c058bc-455d-422c-bf53-f468012c4889 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:34.985546 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c058bc-455d-422c-bf53-f468012c4889 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:34.991050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-66c058bc-455d-422c-bf53-f468012c4889 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:34.991050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:34.991050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:34.991050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:34.991050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:34.991050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:34.991050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:34.991050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:34.991050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:34.991050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:34.991050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:34.991050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:34.991050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:34.991050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:34.991050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:34.991050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:34.991946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:34.991946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:34.991946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:34.991946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:34.991946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:34.991946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:34.991946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:34.991946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:34.991946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:34.991946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:34.991946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:34.991946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:34.991946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:34.991946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:34.991946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:34.991946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:34.992834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:34.992834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:34.992834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:34.992834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:34.992834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:34.992834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:34.992834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:34.992834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:34.992834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:34.992834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:34.992834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:34.992834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:34.995354 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 276/554] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:43:34 2022] GET /identity/v3/roles/service => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:35.001962 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c058bc-455d-422c-bf53-f468012c4889 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:35.161158 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c058bc-455d-422c-bf53-f468012c4889 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:35.163855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f659e18-c49f-4ec9-998a-817371408e87 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:35.207951 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef35f064-ae72-4037-a037-64b966ed5683 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:35.208286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef35f064-ae72-4037-a037-64b966ed5683 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:35.208549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef35f064-ae72-4037-a037-64b966ed5683 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:35.208822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef35f064-ae72-4037-a037-64b966ed5683 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:35.209489 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef35f064-ae72-4037-a037-64b966ed5683 None admin] RBAC: Adding query filter params (name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:35.209844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef35f064-ae72-4037-a037-64b966ed5683 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:35.211426 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef35f064-ae72-4037-a037-64b966ed5683 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:35.222871 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 277/555] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:43:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:35.223718 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef35f064-ae72-4037-a037-64b966ed5683 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:35.246228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8db87db-790b-43af-b614-c4482824f64d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:35.247253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8db87db-790b-43af-b614-c4482824f64d None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:35.247778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8db87db-790b-43af-b614-c4482824f64d None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:35.248249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8db87db-790b-43af-b614-c4482824f64d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:35.277047 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:35.282853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:35.335744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8db87db-790b-43af-b614-c4482824f64d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:35.337380 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8db87db-790b-43af-b614-c4482824f64d None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:35.355752 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef35f064-ae72-4037-a037-64b966ed5683 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:35.393158 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f16012-6676-4dbf-b401-ce5f44cd1506 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:35.394141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f16012-6676-4dbf-b401-ce5f44cd1506 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:35.394723 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f16012-6676-4dbf-b401-ce5f44cd1506 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:35.395266 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f16012-6676-4dbf-b401-ce5f44cd1506 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:35.407934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f16012-6676-4dbf-b401-ce5f44cd1506 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:35.415676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-02f16012-6676-4dbf-b401-ce5f44cd1506 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:35.415676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:35.415676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:35.415676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:35.415676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:35.415676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:35.415676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:35.415676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:35.415676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:35.415676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:35.415676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:35.415676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:35.415676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:35.415676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:35.415676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:35.415676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:35.416670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:35.416670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:35.416670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:35.416670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:35.416670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:35.416670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:35.416670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:35.416670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:35.416670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:35.416670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:35.416670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:35.416670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:35.416670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:35.416670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:35.416670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:35.416670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:35.418119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:35.418119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:35.418119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:35.418119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:35.418119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:35.418119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:35.418119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:35.418119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:35.418119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:35.418119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:35.418119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:35.418119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:35.418119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 278/556] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:43:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:35.418119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f16012-6676-4dbf-b401-ce5f44cd1506 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:35.474598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 279/557] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 549 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:35.476274 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8db87db-790b-43af-b614-c4482824f64d None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:35.538762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:35.551716 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:35.557659 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f16012-6676-4dbf-b401-ce5f44cd1506 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:35.590986 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8db87db-790b-43af-b614-c4482824f64d None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:35.597714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7cfc1e-c5ab-4853-924e-0a2f1c681f2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:35.598374 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7cfc1e-c5ab-4853-924e-0a2f1c681f2d None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:35.598761 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7cfc1e-c5ab-4853-924e-0a2f1c681f2d None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:35.599036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7cfc1e-c5ab-4853-924e-0a2f1c681f2d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:35.609756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7cfc1e-c5ab-4853-924e-0a2f1c681f2d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:35.611224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7cfc1e-c5ab-4853-924e-0a2f1c681f2d None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:35.626511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-2d7cfc1e-c5ab-4853-924e-0a2f1c681f2d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:35.626511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:35.626511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:35.626511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:35.626511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:35.626511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:35.626511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:35.626511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:35.626511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:35.626511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:35.626511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:35.626511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:35.626511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:35.626511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:35.626511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:35.626511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:35.627523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:35.627523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:35.627523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:35.627523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:35.627523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:35.627523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:35.627523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:35.627523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:35.627523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:35.627523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:35.627523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:35.627523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:35.627523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:35.627523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:35.627523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:35.627523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:35.627523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:35.628467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:35.628467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:35.628467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:35.630769 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb2aa1b-933a-4852-bf33-6e78feb2ca76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:35.631383 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb2aa1b-933a-4852-bf33-6e78feb2ca76 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:35.632303 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb2aa1b-933a-4852-bf33-6e78feb2ca76 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:35.632732 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 279/558] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:35.633059 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb2aa1b-933a-4852-bf33-6e78feb2ca76 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:35.633540 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7cfc1e-c5ab-4853-924e-0a2f1c681f2d None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:35.633736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb2aa1b-933a-4852-bf33-6e78feb2ca76 None admin] RBAC: Adding query filter params (name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:35.634040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb2aa1b-933a-4852-bf33-6e78feb2ca76 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:35.635432 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb2aa1b-933a-4852-bf33-6e78feb2ca76 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:35.648850 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 280/559] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:43:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:35.648850 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175d30b4-509e-470f-9341-321e0b1c3502 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:35.648850 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175d30b4-509e-470f-9341-321e0b1c3502 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:35.648850 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175d30b4-509e-470f-9341-321e0b1c3502 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:35.648850 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 281/560] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:35.648850 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175d30b4-509e-470f-9341-321e0b1c3502 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:35.748844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7cfc1e-c5ab-4853-924e-0a2f1c681f2d None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:35.761915 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175d30b4-509e-470f-9341-321e0b1c3502 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:35.779348 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:35.785975 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f62401-dc98-4007-8467-a99f91ce7bba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:35.786877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f62401-dc98-4007-8467-a99f91ce7bba None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:35.787360 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f62401-dc98-4007-8467-a99f91ce7bba None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:35.788301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f62401-dc98-4007-8467-a99f91ce7bba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:35.797327 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81b8bae-5e1e-4c15-bd9a-e2dc2b77e3c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:35.797930 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81b8bae-5e1e-4c15-bd9a-e2dc2b77e3c8 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:35.798207 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81b8bae-5e1e-4c15-bd9a-e2dc2b77e3c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:35.798508 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a81b8bae-5e1e-4c15-bd9a-e2dc2b77e3c8 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:35.799153 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81b8bae-5e1e-4c15-bd9a-e2dc2b77e3c8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:35.799453 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81b8bae-5e1e-4c15-bd9a-e2dc2b77e3c8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:35.800957 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a81b8bae-5e1e-4c15-bd9a-e2dc2b77e3c8 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:35.801445 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f62401-dc98-4007-8467-a99f91ce7bba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:35.803155 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f62401-dc98-4007-8467-a99f91ce7bba None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:35.817806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 282/561] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:35.819068 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81b8bae-5e1e-4c15-bd9a-e2dc2b77e3c8 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:35.822970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-41f62401-dc98-4007-8467-a99f91ce7bba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:35.822970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:35.822970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:35.822970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:35.822970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:35.822970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:35.822970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:35.822970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:35.822970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:35.822970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:35.822970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:35.822970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:35.822970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:35.822970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:35.822970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:35.822970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:35.824082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:35.824082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:35.824082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:35.824082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:35.824082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:35.824082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:35.824082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:35.824082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:35.824082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:35.824082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:35.824082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:35.824082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:35.824082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:35.824082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:35.824082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:35.824082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:35.824082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:35.825102 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:35.825102 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:35.825102 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:35.825102 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 280/562] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:35.825484 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d0d39c-1ee1-4cde-a11e-6b7b0e91bceb None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:35.825743 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d0d39c-1ee1-4cde-a11e-6b7b0e91bceb None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:35.826013 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d0d39c-1ee1-4cde-a11e-6b7b0e91bceb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:35.834367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:35.846178 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-98d0d39c-1ee1-4cde-a11e-6b7b0e91bceb None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:35.847537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-98d0d39c-1ee1-4cde-a11e-6b7b0e91bceb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:35.920955 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 281/563] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:35.926573 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d0d39c-1ee1-4cde-a11e-6b7b0e91bceb None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:35.940538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a81b8bae-5e1e-4c15-bd9a-e2dc2b77e3c8 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:35.979440 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd006e7-8d70-4f14-9d65-6d80ea03420c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:35.980183 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd006e7-8d70-4f14-9d65-6d80ea03420c None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:35.980534 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd006e7-8d70-4f14-9d65-6d80ea03420c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:35.980872 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd006e7-8d70-4f14-9d65-6d80ea03420c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:35.991759 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd006e7-8d70-4f14-9d65-6d80ea03420c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:35.993204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd006e7-8d70-4f14-9d65-6d80ea03420c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:36.010451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-fcd006e7-8d70-4f14-9d65-6d80ea03420c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:36.010451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:36.010451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:36.010451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:36.010451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:36.010451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:36.010451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:36.010451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:36.010451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:36.010451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:36.010451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:36.010451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:36.010451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:36.010451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:36.010451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:36.010451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:36.011362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:36.011362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:36.011362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:36.011362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:36.011362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:36.011362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:36.011362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:36.011362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:36.011362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:36.011362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:36.011362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:36.011362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:36.011362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:36.011362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:36.011362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:36.011362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:36.011362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:36.012233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:36.012233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:36.012233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:36.012233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 283/564] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:36.012765 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd006e7-8d70-4f14-9d65-6d80ea03420c None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:36.051012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:36.054599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d0d39c-1ee1-4cde-a11e-6b7b0e91bceb None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:36.086079 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:36.091918 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4ec82f-b117-4151-ae9d-5e53ed367e04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:36.092651 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4ec82f-b117-4151-ae9d-5e53ed367e04 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:36.092991 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4ec82f-b117-4151-ae9d-5e53ed367e04 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:36.093262 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4ec82f-b117-4151-ae9d-5e53ed367e04 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:36.100989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4ec82f-b117-4151-ae9d-5e53ed367e04 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:36.102594 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4ec82f-b117-4151-ae9d-5e53ed367e04 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:36.116821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-fc4ec82f-b117-4151-ae9d-5e53ed367e04 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 23 13:43:36.116821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:36.116821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:36.116821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:36.116821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:36.116821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:36.116821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:36.116821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:36.116821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:36.116821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:36.116821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:36.116821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:36.116821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:43:36.116821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:43:36.116821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:43:36.116821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:43:36.117928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:36.117928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:36.117928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:43:36.117928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:36.117928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:43:36.117928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:36.117928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:36.117928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:36.117928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:36.117928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:36.117928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:36.117928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:36.117928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:36.117928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:36.117928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:36.117928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:36.117928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:36.117928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:36.118899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:36.118899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:36.118899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:43:36.118899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:43:36.118899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:43:36.118899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:43:36.118899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:43:36.118899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:43:36.118899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 23 13:43:36.118899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:36.118899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 282/565] 10.222.0.62 () {56 vars in 1093 bytes} [Fri Sep 23 13:43:35 2022] GET /identity/v3/users/nova => generated 82 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:36.118899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4ec82f-b117-4151-ae9d-5e53ed367e04 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:36.129864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd006e7-8d70-4f14-9d65-6d80ea03420c None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:36.165474 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a13114e-b0cc-47d0-aa02-47e17a0f8d1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:36.166249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a13114e-b0cc-47d0-aa02-47e17a0f8d1c None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:36.166565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a13114e-b0cc-47d0-aa02-47e17a0f8d1c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:36.166835 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a13114e-b0cc-47d0-aa02-47e17a0f8d1c None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:36.167475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a13114e-b0cc-47d0-aa02-47e17a0f8d1c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:36.167774 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a13114e-b0cc-47d0-aa02-47e17a0f8d1c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:36.169126 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a13114e-b0cc-47d0-aa02-47e17a0f8d1c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:36.181704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 284/566] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:36.182962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03aa0b2-f405-446e-a026-30bf95b549b4 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:36.183240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03aa0b2-f405-446e-a026-30bf95b549b4 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:36.183508 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03aa0b2-f405-446e-a026-30bf95b549b4 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:36.186459 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 285/567] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:36 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:36.186459 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03aa0b2-f405-446e-a026-30bf95b549b4 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:36.237012 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4ec82f-b117-4151-ae9d-5e53ed367e04 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:36.273921 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675c760d-81fc-4940-9f10-46ad0fe4838d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:36.274475 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675c760d-81fc-4940-9f10-46ad0fe4838d None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:36.274751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675c760d-81fc-4940-9f10-46ad0fe4838d None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:36.275016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675c760d-81fc-4940-9f10-46ad0fe4838d None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:36.275647 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675c760d-81fc-4940-9f10-46ad0fe4838d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:36.275962 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675c760d-81fc-4940-9f10-46ad0fe4838d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:36.277295 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-675c760d-81fc-4940-9f10-46ad0fe4838d None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:36.279511 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:36.293688 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 283/568] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:36.293688 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675c760d-81fc-4940-9f10-46ad0fe4838d None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:36.309241 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03aa0b2-f405-446e-a026-30bf95b549b4 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:36.326671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44598992-00a3-4c0b-a1cb-400b142c73b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:36.327273 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44598992-00a3-4c0b-a1cb-400b142c73b1 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:36.327547 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44598992-00a3-4c0b-a1cb-400b142c73b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:36.327813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44598992-00a3-4c0b-a1cb-400b142c73b1 None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:36.328532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44598992-00a3-4c0b-a1cb-400b142c73b1 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:36.328833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44598992-00a3-4c0b-a1cb-400b142c73b1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:36.330238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44598992-00a3-4c0b-a1cb-400b142c73b1 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:36.346777 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 286/569] 10.222.0.62 () {56 vars in 1133 bytes} [Fri Sep 23 13:43:36 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:36.347687 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:36.347927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f84ecb-eae2-4f12-98ef-2e6458bc41ab None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:36.348207 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f84ecb-eae2-4f12-98ef-2e6458bc41ab None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:36.348471 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f84ecb-eae2-4f12-98ef-2e6458bc41ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:36.366192 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-93f84ecb-eae2-4f12-98ef-2e6458bc41ab None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:36.367462 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-93f84ecb-eae2-4f12-98ef-2e6458bc41ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:36.395382 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-675c760d-81fc-4940-9f10-46ad0fe4838d None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:36.429237 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf0e1ac-dfe8-493d-9329-e5d1fed59c99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:36.429857 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf0e1ac-dfe8-493d-9329-e5d1fed59c99 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:36.430138 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf0e1ac-dfe8-493d-9329-e5d1fed59c99 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:36.430461 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf0e1ac-dfe8-493d-9329-e5d1fed59c99 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:36.432399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 287/570] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:36.433226 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f84ecb-eae2-4f12-98ef-2e6458bc41ab None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:36.448349 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf0e1ac-dfe8-493d-9329-e5d1fed59c99 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:36.449790 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adf0e1ac-dfe8-493d-9329-e5d1fed59c99 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:36.461182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-adf0e1ac-dfe8-493d-9329-e5d1fed59c99 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:36.461182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:36.461182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:36.461182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:36.461182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:36.461182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:36.461182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:36.461182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:36.461182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:36.461182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:36.461182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:36.461182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:36.461182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:43:36.461182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:43:36.461182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:43:36.462067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:43:36.462067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:36.462067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:36.462067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:36.462067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:36.462067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:36.462067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:36.462067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:36.462067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:36.462067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:36.462067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:36.462067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:36.462067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:36.462067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:36.462067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:36.462067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:36.462067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:36.463286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:43:36.463286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:43:36.463286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:43:36.463286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:43:36.463286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:43:36.463286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:43:36.463286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:36.463286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:36.463286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 284/571] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:36.463286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf0e1ac-dfe8-493d-9329-e5d1fed59c99 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:36.520437 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:36.543773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f84ecb-eae2-4f12-98ef-2e6458bc41ab None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:36.568648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf0e1ac-dfe8-493d-9329-e5d1fed59c99 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:36.578510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9885e8c4-adb9-4e8b-96b2-4521ca93dae3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:36.579125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9885e8c4-adb9-4e8b-96b2-4521ca93dae3 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:36.579400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9885e8c4-adb9-4e8b-96b2-4521ca93dae3 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:36.579669 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9885e8c4-adb9-4e8b-96b2-4521ca93dae3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:36.582620 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:36.590244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9885e8c4-adb9-4e8b-96b2-4521ca93dae3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:36.595297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-9885e8c4-adb9-4e8b-96b2-4521ca93dae3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:36.595297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:36.595297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:36.595297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:36.595297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:36.595297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:36.595297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:36.595297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:36.595297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:36.595297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:36.595297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:36.595297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:36.595297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:36.595297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:36.595297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:36.595297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:36.596365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:36.596365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:36.596365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:36.596365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:36.596365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:36.596365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:36.596365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:36.596365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:36.596365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:36.596365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:36.596365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:36.596365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:36.596365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:36.596365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:36.596365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:36.596365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:36.597292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:36.597292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:36.597292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:36.597292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:36.597292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:36.597292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:36.597292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:36.597292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:36.597292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:36.597292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:36.597292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:36.597292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:36.599569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 288/572] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:43:36 2022] GET /identity/v3/roles/service => generated 85 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:36.601110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9885e8c4-adb9-4e8b-96b2-4521ca93dae3 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:36.607981 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e7f309-e8d8-4da8-9a20-c0a1e66f3ccf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:36.608855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e7f309-e8d8-4da8-9a20-c0a1e66f3ccf None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:36.609336 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e7f309-e8d8-4da8-9a20-c0a1e66f3ccf None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:36.609877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e7f309-e8d8-4da8-9a20-c0a1e66f3ccf None admin] PATH_INFO: `/v3/users/glance` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:36.619126 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e7f309-e8d8-4da8-9a20-c0a1e66f3ccf None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:36.620994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e7f309-e8d8-4da8-9a20-c0a1e66f3ccf None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:36.632582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-45e7f309-e8d8-4da8-9a20-c0a1e66f3ccf None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 23 13:43:36.632582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:36.632582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:36.632582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:36.632582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:36.632582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:36.632582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:36.632582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:36.632582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:36.632582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:36.632582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:36.632582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:36.632582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:43:36.632582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:43:36.632582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:43:36.632582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:43:36.633673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:36.633673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:36.633673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:43:36.633673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:36.633673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:43:36.633673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:36.633673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:36.633673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:36.633673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:36.633673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:36.633673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:36.633673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:36.633673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:36.633673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:36.633673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:36.633673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:36.633673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:36.633673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:36.634636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:36.634636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:36.634636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:43:36.634636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:43:36.634636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:43:36.634636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:43:36.634636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:43:36.634636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:43:36.634636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 23 13:43:36.634636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:36.634636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 285/573] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:36 2022] GET /identity/v3/users/glance => generated 84 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:36.634636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e7f309-e8d8-4da8-9a20-c0a1e66f3ccf None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:36.707925 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9885e8c4-adb9-4e8b-96b2-4521ca93dae3 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:36.739929 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e7f309-e8d8-4da8-9a20-c0a1e66f3ccf None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:36.743164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1b5fe7-95ab-4019-b047-d00bf757e567 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:36.743894 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1b5fe7-95ab-4019-b047-d00bf757e567 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:36.744064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1b5fe7-95ab-4019-b047-d00bf757e567 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:36.744329 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1b5fe7-95ab-4019-b047-d00bf757e567 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:36.747335 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:36.754910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1b5fe7-95ab-4019-b047-d00bf757e567 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:36.758454 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1b5fe7-95ab-4019-b047-d00bf757e567 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:36.770789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-5d1b5fe7-95ab-4019-b047-d00bf757e567 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:36.770789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:36.770789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:36.770789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:36.770789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:36.770789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:36.770789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:36.770789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:36.770789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:36.770789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:36.770789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:36.770789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:36.770789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:36.770789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:36.770789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:36.770789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:36.771725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:36.771725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:36.771725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:36.771725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:36.771725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:36.771725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:36.771725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:36.771725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:36.771725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:36.771725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:36.771725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:36.771725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:36.771725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:36.771725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:36.771725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:36.771725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:36.771725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:36.772630 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:36.772630 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:36.772630 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:36.772630 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 289/574] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:36.772857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1b5fe7-95ab-4019-b047-d00bf757e567 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:36.779816 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad0d6f8-bf5b-40e9-9abf-97384ee19454 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:36.780422 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad0d6f8-bf5b-40e9-9abf-97384ee19454 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:36.780697 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad0d6f8-bf5b-40e9-9abf-97384ee19454 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:36.780969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad0d6f8-bf5b-40e9-9abf-97384ee19454 None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:36.781652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad0d6f8-bf5b-40e9-9abf-97384ee19454 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:36.781948 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad0d6f8-bf5b-40e9-9abf-97384ee19454 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:36.783670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad0d6f8-bf5b-40e9-9abf-97384ee19454 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:36.796935 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 286/575] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:43:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:36.797655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad0d6f8-bf5b-40e9-9abf-97384ee19454 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:36.828843 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:36.876766 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1b5fe7-95ab-4019-b047-d00bf757e567 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:36.895225 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad0d6f8-bf5b-40e9-9abf-97384ee19454 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:36.909886 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42406284-20a8-473f-af0d-f8cd18d634af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:36.910516 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42406284-20a8-473f-af0d-f8cd18d634af None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:36.910795 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42406284-20a8-473f-af0d-f8cd18d634af None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:36.911064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42406284-20a8-473f-af0d-f8cd18d634af None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:36.911727 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42406284-20a8-473f-af0d-f8cd18d634af None admin] RBAC: Adding query filter params (name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:36.912023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42406284-20a8-473f-af0d-f8cd18d634af None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:36.914055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42406284-20a8-473f-af0d-f8cd18d634af None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:36.922056 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 290/576] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:43:36 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:36.922863 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42406284-20a8-473f-af0d-f8cd18d634af None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:36.929510 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26721466-c338-4466-85b7-ed70ea1c9cec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:36.930106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26721466-c338-4466-85b7-ed70ea1c9cec None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:36.930404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26721466-c338-4466-85b7-ed70ea1c9cec None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:36.930672 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26721466-c338-4466-85b7-ed70ea1c9cec None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:36.931389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26721466-c338-4466-85b7-ed70ea1c9cec None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:36.931687 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26721466-c338-4466-85b7-ed70ea1c9cec None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:36.933083 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26721466-c338-4466-85b7-ed70ea1c9cec None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:36.949525 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 287/577] 10.222.0.62 () {56 vars in 1137 bytes} [Fri Sep 23 13:43:36 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:36.950303 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26721466-c338-4466-85b7-ed70ea1c9cec None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:36.975905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:37.025158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42406284-20a8-473f-af0d-f8cd18d634af None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:37.039754 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:37.050419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26721466-c338-4466-85b7-ed70ea1c9cec None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:37.063133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f66abd-1af4-466c-a582-78f3d7f52830 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:37.063760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f66abd-1af4-466c-a582-78f3d7f52830 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:37.064041 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f66abd-1af4-466c-a582-78f3d7f52830 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:37.064313 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f66abd-1af4-466c-a582-78f3d7f52830 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:37.074764 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f66abd-1af4-466c-a582-78f3d7f52830 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:37.080116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-21f66abd-1af4-466c-a582-78f3d7f52830 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:37.080116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:37.080116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:37.080116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:37.080116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:37.080116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:37.080116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:37.080116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:37.080116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:37.080116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:37.080116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:37.080116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:37.080116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:37.080116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:37.080116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:37.080116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:37.081168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:37.081168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:37.081168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:37.081168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:37.081168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:37.081168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:37.081168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:37.081168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:37.081168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:37.081168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:37.081168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:37.081168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:37.081168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:37.081168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:37.081168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:37.081168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:37.082159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:37.082159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:37.082159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:37.082159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:37.082159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:37.082159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:37.082159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:37.082159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:37.082159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:37.082159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:37.082159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:37.082159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:37.082159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 291/578] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:43:36 2022] GET /identity/v3/roles/service => generated 85 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:37.082159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f66abd-1af4-466c-a582-78f3d7f52830 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:37.083603 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4c5e16-6315-485c-b6f9-c7bbabacf45c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:37.084204 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4c5e16-6315-485c-b6f9-c7bbabacf45c None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:37.085030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4c5e16-6315-485c-b6f9-c7bbabacf45c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:37.085370 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4c5e16-6315-485c-b6f9-c7bbabacf45c None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:37.086098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4c5e16-6315-485c-b6f9-c7bbabacf45c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:37.086426 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4c5e16-6315-485c-b6f9-c7bbabacf45c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:37.087797 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4c5e16-6315-485c-b6f9-c7bbabacf45c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:37.105776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 288/579] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:37.105776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4c5e16-6315-485c-b6f9-c7bbabacf45c None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:37.187242 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f66abd-1af4-466c-a582-78f3d7f52830 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:37.197277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:37.210026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4c5e16-6315-485c-b6f9-c7bbabacf45c None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:37.220468 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a84c25f-4ec0-4653-8f97-2a23cd797589 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:37.221069 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a84c25f-4ec0-4653-8f97-2a23cd797589 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:37.221348 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a84c25f-4ec0-4653-8f97-2a23cd797589 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:37.221617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a84c25f-4ec0-4653-8f97-2a23cd797589 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:37.222360 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a84c25f-4ec0-4653-8f97-2a23cd797589 None admin] RBAC: Adding query filter params (role.id=809612fc6d62442cbcc179b5c4123465, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=4b9269509f1d454eab7a3008c2ba0039) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:37.222682 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a84c25f-4ec0-4653-8f97-2a23cd797589 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:37.224103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a84c25f-4ec0-4653-8f97-2a23cd797589 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:37.230081 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 292/580] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:43:37 2022] GET /identity/v3/role_assignments?user.id=4b9269509f1d454eab7a3008c2ba0039&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=809612fc6d62442cbcc179b5c4123465 => generated 259 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:37.231004 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a84c25f-4ec0-4653-8f97-2a23cd797589 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:37.245296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b2d2c7-1b6c-4e9b-8eea-51ae247e66bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:37.245994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b2d2c7-1b6c-4e9b-8eea-51ae247e66bc None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:37.246226 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b2d2c7-1b6c-4e9b-8eea-51ae247e66bc None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:37.246526 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b2d2c7-1b6c-4e9b-8eea-51ae247e66bc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:37.262173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b2d2c7-1b6c-4e9b-8eea-51ae247e66bc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:37.264113 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b2d2c7-1b6c-4e9b-8eea-51ae247e66bc None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:37.280399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-15b2d2c7-1b6c-4e9b-8eea-51ae247e66bc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:37.280399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:37.280399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:37.280399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:37.280399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:37.280399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:37.280399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:37.280399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:37.280399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:37.280399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:37.280399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:37.280399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:37.280399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:37.280399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:37.280399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:37.280399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:37.281502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:37.281502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:37.281502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:37.281502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:37.281502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:37.281502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:37.281502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:37.281502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:37.281502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:37.281502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:37.281502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:37.281502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:37.281502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:37.281502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:37.281502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:37.281502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:37.281502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:37.282409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:37.282409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:37.282409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:37.282409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 289/581] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:37.282409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b2d2c7-1b6c-4e9b-8eea-51ae247e66bc None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:37.288921 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:37.353729 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a84c25f-4ec0-4653-8f97-2a23cd797589 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:37.390487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d22af0e-9530-405d-ada9-ff05901be471 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:37.391118 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d22af0e-9530-405d-ada9-ff05901be471 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:37.391410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d22af0e-9530-405d-ada9-ff05901be471 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:37.391660 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d22af0e-9530-405d-ada9-ff05901be471 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:37.403937 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d22af0e-9530-405d-ada9-ff05901be471 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:37.405422 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d22af0e-9530-405d-ada9-ff05901be471 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:37.413165 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b2d2c7-1b6c-4e9b-8eea-51ae247e66bc None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:37.420514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-9d22af0e-9530-405d-ada9-ff05901be471 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:37.420514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:37.420514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:37.420514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:37.420514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:37.420514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:37.420514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:37.420514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:37.420514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:37.420514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:37.420514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:37.420514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:37.420514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:37.420514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:37.420514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:37.420514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:37.421483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:37.421483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:37.421483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:37.421483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:37.421483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:37.421483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:37.421483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:37.421483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:37.421483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:37.421483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:37.421483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:37.421483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:37.421483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:37.421483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:37.421483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:37.421483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:37.421483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:37.422437 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:37.422437 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:37.422437 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:37.422437 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 293/582] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:37.423329 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d22af0e-9530-405d-ada9-ff05901be471 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:37.456167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:37.457111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0239ed5-5c17-44a2-9b63-a9def7d805a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:37.457714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0239ed5-5c17-44a2-9b63-a9def7d805a2 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:37.457994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0239ed5-5c17-44a2-9b63-a9def7d805a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:37.458269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0239ed5-5c17-44a2-9b63-a9def7d805a2 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:37.458959 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0239ed5-5c17-44a2-9b63-a9def7d805a2 None admin] RBAC: Adding query filter params (name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:37.459257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0239ed5-5c17-44a2-9b63-a9def7d805a2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:37.461861 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0239ed5-5c17-44a2-9b63-a9def7d805a2 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:37.474700 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 290/583] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:43:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:37.475856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0239ed5-5c17-44a2-9b63-a9def7d805a2 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:37.539042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d22af0e-9530-405d-ada9-ff05901be471 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:37.540994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:37.576123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5ad0e4-c423-47be-956a-f6ad419f8d32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:37.577019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5ad0e4-c423-47be-956a-f6ad419f8d32 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:37.577438 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5ad0e4-c423-47be-956a-f6ad419f8d32 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:37.577955 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5ad0e4-c423-47be-956a-f6ad419f8d32 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:37.581837 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0239ed5-5c17-44a2-9b63-a9def7d805a2 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:37.587081 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5ad0e4-c423-47be-956a-f6ad419f8d32 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:37.588574 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5ad0e4-c423-47be-956a-f6ad419f8d32 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:37.601826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-1d5ad0e4-c423-47be-956a-f6ad419f8d32 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:37.601826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:37.601826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:37.601826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:37.601826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:37.601826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:37.601826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:37.601826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:37.601826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:37.601826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:37.601826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:37.601826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:37.601826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:43:37.601826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:43:37.601826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:43:37.602771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:43:37.602771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:37.602771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:37.602771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:37.602771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:37.602771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:37.602771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:37.602771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:37.602771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:37.602771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:37.602771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:37.602771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:37.602771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:37.602771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:37.602771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:37.602771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:37.602771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:37.603635 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:43:37.603635 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:43:37.603635 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:43:37.603635 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:43:37.603635 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:43:37.603635 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:43:37.603635 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:37.603635 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:37.603635 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 294/584] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:37.604172 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5ad0e4-c423-47be-956a-f6ad419f8d32 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:37.624393 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def2d1d9-c006-423d-ae68-cbd014410578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:37.625314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def2d1d9-c006-423d-ae68-cbd014410578 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:37.625799 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def2d1d9-c006-423d-ae68-cbd014410578 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:37.626712 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def2d1d9-c006-423d-ae68-cbd014410578 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:37.627416 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def2d1d9-c006-423d-ae68-cbd014410578 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:37.627736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def2d1d9-c006-423d-ae68-cbd014410578 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:37.629126 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def2d1d9-c006-423d-ae68-cbd014410578 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:37.645854 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 291/585] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:37.645854 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def2d1d9-c006-423d-ae68-cbd014410578 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:37.700168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:37.715635 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5ad0e4-c423-47be-956a-f6ad419f8d32 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:37.753447 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9a18af-829a-4bfd-b9a7-f6776ef09a02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:37.754074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9a18af-829a-4bfd-b9a7-f6776ef09a02 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:37.754378 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9a18af-829a-4bfd-b9a7-f6776ef09a02 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:37.754649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf9a18af-829a-4bfd-b9a7-f6776ef09a02 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:37.755297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9a18af-829a-4bfd-b9a7-f6776ef09a02 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:37.755595 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9a18af-829a-4bfd-b9a7-f6776ef09a02 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:37.757636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf9a18af-829a-4bfd-b9a7-f6776ef09a02 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:37.762579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def2d1d9-c006-423d-ae68-cbd014410578 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:37.773798 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 295/586] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:37.773798 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9a18af-829a-4bfd-b9a7-f6776ef09a02 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:37.792667 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:37.805287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a93b41-55f0-49d8-aaa9-56fbf7fe51f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:37.809652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a93b41-55f0-49d8-aaa9-56fbf7fe51f1 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:37.809652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a93b41-55f0-49d8-aaa9-56fbf7fe51f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:37.809652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a93b41-55f0-49d8-aaa9-56fbf7fe51f1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:37.818046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a93b41-55f0-49d8-aaa9-56fbf7fe51f1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:37.820057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a93b41-55f0-49d8-aaa9-56fbf7fe51f1 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:37.835309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-d0a93b41-55f0-49d8-aaa9-56fbf7fe51f1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:37.835309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:37.835309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:37.835309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:37.835309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:37.835309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:37.835309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:37.835309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:37.835309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:37.835309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:37.835309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:37.835309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:37.835309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:37.835309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:37.835309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:37.835309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:37.836243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:37.836243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:37.836243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:37.836243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:37.836243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:37.836243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:37.836243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:37.836243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:37.836243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:37.836243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:37.836243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:37.836243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:37.836243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:37.836243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:37.836243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:37.836243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:37.836243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:37.837111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:37.837111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:37.837111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:37.837111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 292/587] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:37.838101 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a93b41-55f0-49d8-aaa9-56fbf7fe51f1 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:37.882422 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf9a18af-829a-4bfd-b9a7-f6776ef09a02 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:37.918531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2b0e1a-b495-4d54-9874-bf98287d40f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:37.919497 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2b0e1a-b495-4d54-9874-bf98287d40f9 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:37.920010 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2b0e1a-b495-4d54-9874-bf98287d40f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:37.920502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2b0e1a-b495-4d54-9874-bf98287d40f9 None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:37.921399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2b0e1a-b495-4d54-9874-bf98287d40f9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:37.921927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2b0e1a-b495-4d54-9874-bf98287d40f9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:37.923563 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2b0e1a-b495-4d54-9874-bf98287d40f9 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:37.937574 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 296/588] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:43:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:37.938811 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2b0e1a-b495-4d54-9874-bf98287d40f9 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:37.948553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a93b41-55f0-49d8-aaa9-56fbf7fe51f1 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:37.949340 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:37.985282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15b685c-341d-4418-839e-78577d742823 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:37.986074 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15b685c-341d-4418-839e-78577d742823 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:37.986388 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15b685c-341d-4418-839e-78577d742823 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:37.986654 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15b685c-341d-4418-839e-78577d742823 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:37.995485 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15b685c-341d-4418-839e-78577d742823 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:37.996900 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a15b685c-341d-4418-839e-78577d742823 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:38.009477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-a15b685c-341d-4418-839e-78577d742823 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 23 13:43:38.009477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:38.009477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:38.009477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:38.009477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:38.009477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:38.009477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:38.009477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:38.009477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:38.009477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:38.009477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:38.009477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:38.009477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:43:38.009477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:43:38.009477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:43:38.009477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:43:38.010621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:38.010621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:38.010621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:43:38.010621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:38.010621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:43:38.010621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:38.010621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:38.010621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:38.010621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:38.010621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:38.010621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:38.010621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:38.010621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:38.010621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:38.010621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:38.010621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:38.010621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:38.010621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:38.011643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:38.011643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:38.011643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:43:38.011643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:43:38.011643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:43:38.011643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:43:38.011643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:43:38.011643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:43:38.011643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 23 13:43:38.011643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:38.011643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 293/589] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:37 2022] GET /identity/v3/users/cinder => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:38.011643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15b685c-341d-4418-839e-78577d742823 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:38.046110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:38.052404 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2b0e1a-b495-4d54-9874-bf98287d40f9 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:38.098561 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebaee9df-51f1-4fd0-80bc-6fc892a9a7da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:38.099209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaee9df-51f1-4fd0-80bc-6fc892a9a7da None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:38.099488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaee9df-51f1-4fd0-80bc-6fc892a9a7da None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:38.099756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaee9df-51f1-4fd0-80bc-6fc892a9a7da None admin] PATH_INFO: `/v3/projects/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:38.107997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebaee9df-51f1-4fd0-80bc-6fc892a9a7da None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:38.109461 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebaee9df-51f1-4fd0-80bc-6fc892a9a7da None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:38.117285 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15b685c-341d-4418-839e-78577d742823 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:38.123082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-ebaee9df-51f1-4fd0-80bc-6fc892a9a7da None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:38.123082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:38.123082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:38.123082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:38.123082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:38.123082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:38.123082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:38.123082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:38.123082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:38.123082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:38.123082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:38.123082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:38.123082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:43:38.123082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:43:38.123082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:43:38.124106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:43:38.124106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:38.124106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:38.124106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:38.124106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:38.124106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:38.124106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:38.124106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:38.124106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:38.124106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:38.124106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:38.124106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:38.124106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:38.124106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:38.124106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:38.124106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:38.124106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:38.125228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:43:38.125228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:43:38.125228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:43:38.125228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:43:38.125228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:43:38.125228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:43:38.125228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:38.125228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:38.128720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 297/590] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:38.129502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebaee9df-51f1-4fd0-80bc-6fc892a9a7da None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:38.156224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3ad794-f104-4c3d-b953-a05d8bce8b9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:38.157025 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3ad794-f104-4c3d-b953-a05d8bce8b9c None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:38.157453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3ad794-f104-4c3d-b953-a05d8bce8b9c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:38.157876 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3ad794-f104-4c3d-b953-a05d8bce8b9c None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:38.158551 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3ad794-f104-4c3d-b953-a05d8bce8b9c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:38.158855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3ad794-f104-4c3d-b953-a05d8bce8b9c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:38.160316 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3ad794-f104-4c3d-b953-a05d8bce8b9c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:38.173204 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 294/591] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:38.174008 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3ad794-f104-4c3d-b953-a05d8bce8b9c None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:38.206147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:38.250302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebaee9df-51f1-4fd0-80bc-6fc892a9a7da None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:38.283119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:38.285912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d285be2b-2dbb-4492-bbd4-149ed2f85184 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:38.286854 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d285be2b-2dbb-4492-bbd4-149ed2f85184 None admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:38.287288 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d285be2b-2dbb-4492-bbd4-149ed2f85184 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:38.288082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d285be2b-2dbb-4492-bbd4-149ed2f85184 None admin] PATH_INFO: `/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/f01ed94dbb384a178ad0a45a6f3d8c17/roles/809612fc6d62442cbcc179b5c4123465` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:38.289778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3ad794-f104-4c3d-b953-a05d8bce8b9c None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:38.304597 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d285be2b-2dbb-4492-bbd4-149ed2f85184 None admin] RBAC: Authorizing `identity:create_grant(project_id=2a19fd4315ad4a6eb349dfe8cfa66c23, user_id=f01ed94dbb384a178ad0a45a6f3d8c17, role_id=809612fc6d62442cbcc179b5c4123465)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:38.306041 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d285be2b-2dbb-4492-bbd4-149ed2f85184 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:38.325982 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cf7023-ccff-4581-b81c-1c13cb950c94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:38.326612 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cf7023-ccff-4581-b81c-1c13cb950c94 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:38.326885 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cf7023-ccff-4581-b81c-1c13cb950c94 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:38.327147 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cf7023-ccff-4581-b81c-1c13cb950c94 None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:38.327852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cf7023-ccff-4581-b81c-1c13cb950c94 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:38.328148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cf7023-ccff-4581-b81c-1c13cb950c94 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:38.329540 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cf7023-ccff-4581-b81c-1c13cb950c94 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:38.352473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 295/592] 10.222.0.62 () {56 vars in 1137 bytes} [Fri Sep 23 13:43:38 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:38.352473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cf7023-ccff-4581-b81c-1c13cb950c94 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:38.407138 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 298/593] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:43:38 2022] PUT /identity/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/f01ed94dbb384a178ad0a45a6f3d8c17/roles/809612fc6d62442cbcc179b5c4123465 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:43:38.415588 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d285be2b-2dbb-4492-bbd4-149ed2f85184 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:38.475847 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cf7023-ccff-4581-b81c-1c13cb950c94 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:38.515430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaae7918-2b85-4aff-8b21-93ef45540ae7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:38.516043 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaae7918-2b85-4aff-8b21-93ef45540ae7 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:38.516321 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaae7918-2b85-4aff-8b21-93ef45540ae7 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:38.516589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaae7918-2b85-4aff-8b21-93ef45540ae7 None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:38.517468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaae7918-2b85-4aff-8b21-93ef45540ae7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:38.518162 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaae7918-2b85-4aff-8b21-93ef45540ae7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:38.520006 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaae7918-2b85-4aff-8b21-93ef45540ae7 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:38.535831 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 296/594] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:43:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:38.536703 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaae7918-2b85-4aff-8b21-93ef45540ae7 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:38.561443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:38.572928 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d285be2b-2dbb-4492-bbd4-149ed2f85184 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:38.593402 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:38.613759 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554a5918-0c88-42b8-bd14-35698a659c02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:38.614551 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554a5918-0c88-42b8-bd14-35698a659c02 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:38.614717 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554a5918-0c88-42b8-bd14-35698a659c02 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:38.614986 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554a5918-0c88-42b8-bd14-35698a659c02 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:38.624792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554a5918-0c88-42b8-bd14-35698a659c02 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:38.626231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554a5918-0c88-42b8-bd14-35698a659c02 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:38.638702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-554a5918-0c88-42b8-bd14-35698a659c02 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 23 13:43:38.638702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:38.638702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:38.638702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:38.638702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:38.638702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:38.638702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:38.638702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:38.638702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:38.638702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:38.638702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:38.638702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:38.638702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:43:38.638702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:43:38.638702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:43:38.638702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:43:38.639905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:38.639905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:38.639905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:43:38.639905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:38.639905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:43:38.639905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:38.639905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:38.639905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:38.639905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:38.639905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:38.639905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:38.639905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:38.639905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:38.639905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:38.639905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:38.639905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:38.639905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:38.639905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:38.641052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:38.641052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:38.641052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:43:38.641052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:43:38.641052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:43:38.641052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:43:38.641052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:43:38.641052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:43:38.641052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 23 13:43:38.641052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:38.643595 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 299/595] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:43:38 2022] GET /identity/v3/users/neutron => generated 85 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:38.644563 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554a5918-0c88-42b8-bd14-35698a659c02 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:38.661890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaae7918-2b85-4aff-8b21-93ef45540ae7 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:38.709043 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6855bcd4-99f9-462a-8d80-16f930fed648 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:38.711421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6855bcd4-99f9-462a-8d80-16f930fed648 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:38.711805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6855bcd4-99f9-462a-8d80-16f930fed648 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:38.712436 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6855bcd4-99f9-462a-8d80-16f930fed648 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:38.728895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6855bcd4-99f9-462a-8d80-16f930fed648 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:38.730343 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6855bcd4-99f9-462a-8d80-16f930fed648 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:38.755590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-6855bcd4-99f9-462a-8d80-16f930fed648 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:38.755590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:38.755590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:38.755590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:38.755590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:38.755590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:38.755590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:38.755590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:38.755590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:38.755590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:38.755590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:38.755590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:38.755590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:38.755590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:38.755590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:38.755590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:38.757648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:38.757648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:38.757648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:38.757648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:38.757648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:38.757648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:38.757648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:38.757648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:38.757648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:38.757648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:38.757648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:38.757648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:38.757648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:38.757648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:38.757648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:38.757648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:38.757648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:38.758582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:38.758582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:38.758582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:38.760851 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 297/596] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:38.762270 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6855bcd4-99f9-462a-8d80-16f930fed648 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:38.771358 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554a5918-0c88-42b8-bd14-35698a659c02 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:38.812917 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb5b311-65b0-4a47-a622-d0b81607894a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:38.813720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb5b311-65b0-4a47-a622-d0b81607894a None admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:38.814097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb5b311-65b0-4a47-a622-d0b81607894a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:38.814513 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb5b311-65b0-4a47-a622-d0b81607894a None admin] PATH_INFO: `/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/acdf4a7bd608492390176dcc42ca1267/roles/809612fc6d62442cbcc179b5c4123465` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:38.831861 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb5b311-65b0-4a47-a622-d0b81607894a None admin] RBAC: Authorizing `identity:create_grant(project_id=2a19fd4315ad4a6eb349dfe8cfa66c23, user_id=acdf4a7bd608492390176dcc42ca1267, role_id=809612fc6d62442cbcc179b5c4123465)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:38.833572 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb5b311-65b0-4a47-a622-d0b81607894a None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:38.855170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:38.891628 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6855bcd4-99f9-462a-8d80-16f930fed648 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:38.919222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 300/597] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:43:38 2022] PUT /identity/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/acdf4a7bd608492390176dcc42ca1267/roles/809612fc6d62442cbcc179b5c4123465 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:43:38.920914 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb5b311-65b0-4a47-a622-d0b81607894a None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:38.946395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8097f6cb-8e73-463c-be52-a4a8a27f3dc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:38.946395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8097f6cb-8e73-463c-be52-a4a8a27f3dc2 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:38.946395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8097f6cb-8e73-463c-be52-a4a8a27f3dc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:38.946395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8097f6cb-8e73-463c-be52-a4a8a27f3dc2 None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:38.946927 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8097f6cb-8e73-463c-be52-a4a8a27f3dc2 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:38.946927 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8097f6cb-8e73-463c-be52-a4a8a27f3dc2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:38.946927 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8097f6cb-8e73-463c-be52-a4a8a27f3dc2 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:38.980422 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 298/598] 10.222.0.62 () {56 vars in 1139 bytes} [Fri Sep 23 13:43:38 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:38.986392 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8097f6cb-8e73-463c-be52-a4a8a27f3dc2 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:38.989779 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:39.136812 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb5b311-65b0-4a47-a622-d0b81607894a None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:39.175925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8097f6cb-8e73-463c-be52-a4a8a27f3dc2 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:39.206934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a817d84-c856-4130-a6b2-ead356cdd582 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:39.207706 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a817d84-c856-4130-a6b2-ead356cdd582 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:39.208087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a817d84-c856-4130-a6b2-ead356cdd582 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:39.208475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a817d84-c856-4130-a6b2-ead356cdd582 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:39.209370 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a817d84-c856-4130-a6b2-ead356cdd582 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:39.209768 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a817d84-c856-4130-a6b2-ead356cdd582 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:39.211376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a817d84-c856-4130-a6b2-ead356cdd582 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:39.215300 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:39.228199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c236cca9-2c0d-4557-85a1-a60fc5c365dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:39.228970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c236cca9-2c0d-4557-85a1-a60fc5c365dc None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:39.229349 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c236cca9-2c0d-4557-85a1-a60fc5c365dc None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:39.229715 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c236cca9-2c0d-4557-85a1-a60fc5c365dc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:39.231743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 301/599] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:39.232655 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a817d84-c856-4130-a6b2-ead356cdd582 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:39.244060 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c236cca9-2c0d-4557-85a1-a60fc5c365dc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:39.245905 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c236cca9-2c0d-4557-85a1-a60fc5c365dc None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:39.268175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-c236cca9-2c0d-4557-85a1-a60fc5c365dc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:39.268175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:39.268175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:39.268175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:39.268175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:39.268175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:39.268175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:39.268175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:39.268175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:39.268175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:39.268175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:39.268175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:39.268175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:39.268175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:39.268175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:39.268175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:39.270602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:39.270602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:39.270602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:39.270602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:39.270602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:39.270602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:39.270602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:39.270602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:39.270602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:39.270602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:39.270602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:39.270602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:39.270602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:39.270602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:39.270602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:39.270602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:39.270602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:39.273193 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:39.273193 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:39.273193 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:39.276790 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 299/600] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:39.277711 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c236cca9-2c0d-4557-85a1-a60fc5c365dc None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:39.336224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:39.344208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a817d84-c856-4130-a6b2-ead356cdd582 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:39.384013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6696fde-6757-476f-a0f0-86b9c264a1cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:39.384847 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6696fde-6757-476f-a0f0-86b9c264a1cb None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:39.385228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6696fde-6757-476f-a0f0-86b9c264a1cb None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:39.385731 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6696fde-6757-476f-a0f0-86b9c264a1cb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:39.394058 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6696fde-6757-476f-a0f0-86b9c264a1cb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:39.395949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6696fde-6757-476f-a0f0-86b9c264a1cb None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:39.409846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-a6696fde-6757-476f-a0f0-86b9c264a1cb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:39.409846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:39.409846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:39.409846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:39.409846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:39.409846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:39.409846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:39.409846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:39.409846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:39.409846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:39.409846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:39.409846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:39.409846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:43:39.409846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:43:39.409846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:43:39.411009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:43:39.411009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:39.411009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:39.411009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:39.411009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:39.411009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:39.411009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:39.411009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:39.411009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:39.411009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:39.411009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:39.411009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:39.411009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:39.411009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:39.411009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:39.411009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:39.411009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:39.412156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:43:39.412156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:43:39.412156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:43:39.412156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:43:39.412156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:43:39.412156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:43:39.412156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:39.412156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:39.412936 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c236cca9-2c0d-4557-85a1-a60fc5c365dc None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:39.415448 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 302/601] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:39.420050 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6696fde-6757-476f-a0f0-86b9c264a1cb None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:39.450815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9851ff87-2d0f-4338-a04b-daf640d15a45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:39.451348 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9851ff87-2d0f-4338-a04b-daf640d15a45 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:39.451623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9851ff87-2d0f-4338-a04b-daf640d15a45 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:39.451892 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9851ff87-2d0f-4338-a04b-daf640d15a45 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:39.452531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9851ff87-2d0f-4338-a04b-daf640d15a45 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:39.452828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9851ff87-2d0f-4338-a04b-daf640d15a45 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:39.454267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9851ff87-2d0f-4338-a04b-daf640d15a45 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:39.467077 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 300/602] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:39.475606 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9851ff87-2d0f-4338-a04b-daf640d15a45 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:39.535844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:39.563255 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6696fde-6757-476f-a0f0-86b9c264a1cb None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:39.601409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb4d01d-c37a-4d60-b1d3-01435d4ec4c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:39.602865 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb4d01d-c37a-4d60-b1d3-01435d4ec4c6 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:39.603337 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb4d01d-c37a-4d60-b1d3-01435d4ec4c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:39.603745 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb4d01d-c37a-4d60-b1d3-01435d4ec4c6 None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:39.604634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb4d01d-c37a-4d60-b1d3-01435d4ec4c6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:39.605209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb4d01d-c37a-4d60-b1d3-01435d4ec4c6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:39.606934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb4d01d-c37a-4d60-b1d3-01435d4ec4c6 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:39.614205 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9851ff87-2d0f-4338-a04b-daf640d15a45 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:39.621202 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 303/603] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:43:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:39.622790 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3d7ba4-7190-4194-8981-46a438cb59d2 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:39.623240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3d7ba4-7190-4194-8981-46a438cb59d2 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:39.623703 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3d7ba4-7190-4194-8981-46a438cb59d2 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:39.625275 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 304/604] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:39.633803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3d7ba4-7190-4194-8981-46a438cb59d2 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:39.643804 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:39.667405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92555c7d-84b8-47ce-8ecc-c21ec118d60a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:39.669268 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92555c7d-84b8-47ce-8ecc-c21ec118d60a None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:39.669993 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92555c7d-84b8-47ce-8ecc-c21ec118d60a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:39.670425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92555c7d-84b8-47ce-8ecc-c21ec118d60a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:39.680558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92555c7d-84b8-47ce-8ecc-c21ec118d60a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:39.682499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92555c7d-84b8-47ce-8ecc-c21ec118d60a None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:39.699599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-92555c7d-84b8-47ce-8ecc-c21ec118d60a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:39.699599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:39.699599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:39.699599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:39.699599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:39.699599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:39.699599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:39.699599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:39.699599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:39.699599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:39.699599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:39.699599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:39.699599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:43:39.699599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:43:39.699599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:43:39.700655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:43:39.700655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:39.700655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:39.700655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:39.700655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:39.700655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:39.700655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:39.700655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:39.700655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:39.700655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:39.700655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:39.700655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:39.700655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:39.700655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:39.700655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:39.700655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:39.700655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:39.701776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:43:39.701776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:43:39.701776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:43:39.701776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:43:39.701776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:43:39.701776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:43:39.701776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:39.701776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:39.704799 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 301/605] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:39.706104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b5b310-ccd3-4970-aecb-83eb2e833fca None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:39.706537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b5b310-ccd3-4970-aecb-83eb2e833fca None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:39.706930 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b5b310-ccd3-4970-aecb-83eb2e833fca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:39.730896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-92b5b310-ccd3-4970-aecb-83eb2e833fca None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:39.732863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-92b5b310-ccd3-4970-aecb-83eb2e833fca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:39.759204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3d7ba4-7190-4194-8981-46a438cb59d2 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:39.797276 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99995069-62ef-46c7-ae66-fe936b04e081 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:39.798134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99995069-62ef-46c7-ae66-fe936b04e081 None admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:39.798543 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99995069-62ef-46c7-ae66-fe936b04e081 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:39.798907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99995069-62ef-46c7-ae66-fe936b04e081 None admin] PATH_INFO: `/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/fae68ca0754a4b0b88a57f39efa2fb45/roles/809612fc6d62442cbcc179b5c4123465` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:39.816053 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99995069-62ef-46c7-ae66-fe936b04e081 None admin] RBAC: Authorizing `identity:create_grant(project_id=2a19fd4315ad4a6eb349dfe8cfa66c23, user_id=fae68ca0754a4b0b88a57f39efa2fb45, role_id=809612fc6d62442cbcc179b5c4123465)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:39.817687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99995069-62ef-46c7-ae66-fe936b04e081 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:39.824264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 302/606] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:39.831328 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b5b310-ccd3-4970-aecb-83eb2e833fca None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:39.866864 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:39.930297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 305/607] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:43:39 2022] PUT /identity/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/fae68ca0754a4b0b88a57f39efa2fb45/roles/809612fc6d62442cbcc179b5c4123465 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:43:39.959858 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b5b310-ccd3-4970-aecb-83eb2e833fca None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:40.016757 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbeca65d-a6f9-4ff5-a43b-21a8928b1722 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:40.017533 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbeca65d-a6f9-4ff5-a43b-21a8928b1722 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:40.017929 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbeca65d-a6f9-4ff5-a43b-21a8928b1722 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:40.018300 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbeca65d-a6f9-4ff5-a43b-21a8928b1722 None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:40.019114 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbeca65d-a6f9-4ff5-a43b-21a8928b1722 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:40.019518 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbeca65d-a6f9-4ff5-a43b-21a8928b1722 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:40.021079 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbeca65d-a6f9-4ff5-a43b-21a8928b1722 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:40.040960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 303/608] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:43:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:40.047064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99995069-62ef-46c7-ae66-fe936b04e081 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:40.126647 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:40.186031 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99995069-62ef-46c7-ae66-fe936b04e081 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:40.195058 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbeca65d-a6f9-4ff5-a43b-21a8928b1722 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:40.315966 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dee57f2-1ad9-4d80-bb18-993b71b703de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:40.316634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dee57f2-1ad9-4d80-bb18-993b71b703de None admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:40.316909 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dee57f2-1ad9-4d80-bb18-993b71b703de None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:40.317177 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dee57f2-1ad9-4d80-bb18-993b71b703de None admin] PATH_INFO: `/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/580a35627c234a25aa04f5d376177ca5/roles/809612fc6d62442cbcc179b5c4123465` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:40.345480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:40.349353 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dee57f2-1ad9-4d80-bb18-993b71b703de None admin] RBAC: Authorizing `identity:create_grant(project_id=2a19fd4315ad4a6eb349dfe8cfa66c23, user_id=580a35627c234a25aa04f5d376177ca5, role_id=809612fc6d62442cbcc179b5c4123465)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:40.351425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dee57f2-1ad9-4d80-bb18-993b71b703de None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:40.365260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbeca65d-a6f9-4ff5-a43b-21a8928b1722 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:40.401486 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c63d08-89b4-4958-94e0-58a428260b57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:40.402122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c63d08-89b4-4958-94e0-58a428260b57 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:40.402426 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c63d08-89b4-4958-94e0-58a428260b57 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:40.402704 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c63d08-89b4-4958-94e0-58a428260b57 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:40.413977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c63d08-89b4-4958-94e0-58a428260b57 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:40.419308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-07c63d08-89b4-4958-94e0-58a428260b57 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:40.419308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:40.419308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:40.419308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:40.419308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:40.419308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:40.419308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:40.419308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:40.419308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:40.419308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:40.419308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:40.419308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:40.419308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:40.419308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:40.419308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:40.419308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:40.421372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:40.421372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:40.421372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:40.421372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:40.421372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:40.421372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:40.421372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:40.421372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:40.421372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:40.421372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:40.421372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:40.421372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:40.421372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:40.421372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:40.421372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:40.421372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:40.423365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:40.423365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:40.423365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:40.423365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:40.423365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:40.423365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:40.423365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:40.423365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:40.423365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:40.423365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:40.423365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:40.423365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:40.423365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 304/609] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:43:40 2022] GET /identity/v3/roles/service => generated 85 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:43:40.427020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c63d08-89b4-4958-94e0-58a428260b57 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:40.442166 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 306/610] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:43:40 2022] PUT /identity/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/580a35627c234a25aa04f5d376177ca5/roles/809612fc6d62442cbcc179b5c4123465 => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:43:40.583710 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c63d08-89b4-4958-94e0-58a428260b57 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:40.629121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d39c37-d75e-46b0-8510-87203320a539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:40.629739 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d39c37-d75e-46b0-8510-87203320a539 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:40.630022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d39c37-d75e-46b0-8510-87203320a539 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:40.630291 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d39c37-d75e-46b0-8510-87203320a539 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:40.631119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d39c37-d75e-46b0-8510-87203320a539 None admin] RBAC: Adding query filter params (name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:40.631834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d39c37-d75e-46b0-8510-87203320a539 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:40.633512 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d39c37-d75e-46b0-8510-87203320a539 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:40.642018 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 305/611] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:43:40 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:40.648398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dee57f2-1ad9-4d80-bb18-993b71b703de None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:40.791394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dee57f2-1ad9-4d80-bb18-993b71b703de None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:40.799076 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:40.828319 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe85590-ae84-430c-91a1-f13dd5381845 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:40.828990 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe85590-ae84-430c-91a1-f13dd5381845 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:40.829264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe85590-ae84-430c-91a1-f13dd5381845 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:40.829578 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe85590-ae84-430c-91a1-f13dd5381845 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:40.841407 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe85590-ae84-430c-91a1-f13dd5381845 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:40.848029 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe85590-ae84-430c-91a1-f13dd5381845 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:40.871868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-bfe85590-ae84-430c-91a1-f13dd5381845 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:40.871868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:40.871868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:40.871868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:40.871868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:40.871868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:40.871868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:40.871868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:40.871868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:40.871868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:40.871868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:40.871868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:40.871868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:40.871868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:40.871868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:40.871868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:40.872877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:40.872877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:40.872877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:40.872877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:40.872877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:40.872877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:40.872877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:40.872877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:40.872877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:40.872877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:40.872877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:40.872877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:40.872877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:40.872877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:40.872877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:40.872877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:40.872877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:40.874149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:40.874149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:40.874149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:40.874149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 307/612] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:40.880676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d39c37-d75e-46b0-8510-87203320a539 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:40.903065 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:41.019708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a3de05-4ca4-414e-bf96-afd0a0a80d5e None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:41.020414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a3de05-4ca4-414e-bf96-afd0a0a80d5e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:41.020910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a3de05-4ca4-414e-bf96-afd0a0a80d5e None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:41.024019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 308/613] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:41.028356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86c8266-6c03-4ea3-9471-33461ea7d242 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:41.028644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86c8266-6c03-4ea3-9471-33461ea7d242 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:41.028911 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86c8266-6c03-4ea3-9471-33461ea7d242 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:41.048223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c86c8266-6c03-4ea3-9471-33461ea7d242 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:41.049107 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d39c37-d75e-46b0-8510-87203320a539 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:41.049407 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c86c8266-6c03-4ea3-9471-33461ea7d242 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:41.118154 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bad9b7-50c9-417a-8310-5afaf0d8d197 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:41.119266 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bad9b7-50c9-417a-8310-5afaf0d8d197 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:41.119950 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bad9b7-50c9-417a-8310-5afaf0d8d197 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:41.130436 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bad9b7-50c9-417a-8310-5afaf0d8d197 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:41.130436 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20bad9b7-50c9-417a-8310-5afaf0d8d197 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:41.130436 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20bad9b7-50c9-417a-8310-5afaf0d8d197 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:41.130436 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20bad9b7-50c9-417a-8310-5afaf0d8d197 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:41.147836 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 306/614] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:41.151514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 309/615] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:41.167100 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bad9b7-50c9-417a-8310-5afaf0d8d197 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:41.280594 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:41.325155 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bad9b7-50c9-417a-8310-5afaf0d8d197 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:41.372452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8824601a-66b5-4c98-a739-4ae6bd2ce573 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:41.373011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8824601a-66b5-4c98-a739-4ae6bd2ce573 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:41.373298 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8824601a-66b5-4c98-a739-4ae6bd2ce573 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:41.373550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8824601a-66b5-4c98-a739-4ae6bd2ce573 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:41.387462 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8824601a-66b5-4c98-a739-4ae6bd2ce573 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:41.388930 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8824601a-66b5-4c98-a739-4ae6bd2ce573 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:41.405733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-8824601a-66b5-4c98-a739-4ae6bd2ce573 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 23 13:43:41.405733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:41.405733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:41.405733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:41.405733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:41.405733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:41.405733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:41.405733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:41.405733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:41.405733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:41.405733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:41.405733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:41.405733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:43:41.405733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:43:41.405733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:43:41.405733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:43:41.406781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:41.406781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:41.406781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:43:41.406781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:41.406781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:43:41.406781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:41.406781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:41.406781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:41.406781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:41.406781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:41.406781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:41.406781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:41.406781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:41.406781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:41.406781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:41.406781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:41.406781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:41.406781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:41.407717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:41.407717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:41.407717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:43:41.407717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:43:41.407717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:43:41.407717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:43:41.407717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:43:41.407717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:43:41.407717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 23 13:43:41.407717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:41.407717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 307/616] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:43:41 2022] GET /identity/v3/users/swift => generated 83 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:41.413302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86c8266-6c03-4ea3-9471-33461ea7d242 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:41.520718 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63760ecf-2cbb-4ee7-88ab-2af90a1f14da None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:41.521083 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63760ecf-2cbb-4ee7-88ab-2af90a1f14da None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:41.521246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63760ecf-2cbb-4ee7-88ab-2af90a1f14da None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:41.523734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 308/617] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:41.529507 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7921d2-9294-477f-9daf-3dfee1487d00 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:41.530001 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7921d2-9294-477f-9daf-3dfee1487d00 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:41.530444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b7921d2-9294-477f-9daf-3dfee1487d00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:41.557733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-8b7921d2-9294-477f-9daf-3dfee1487d00 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:41.558957 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8b7921d2-9294-477f-9daf-3dfee1487d00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:41.575873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:41.594231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86c8266-6c03-4ea3-9471-33461ea7d242 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:41.641093 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 309/618] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:41.642152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7921d2-9294-477f-9daf-3dfee1487d00 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:41.643026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a461a1-c73c-46e3-8aac-ac390e3a1604 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:41.643823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a461a1-c73c-46e3-8aac-ac390e3a1604 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:41.644250 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a461a1-c73c-46e3-8aac-ac390e3a1604 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:41.644730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a461a1-c73c-46e3-8aac-ac390e3a1604 None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:41.646060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a461a1-c73c-46e3-8aac-ac390e3a1604 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:41.647276 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a461a1-c73c-46e3-8aac-ac390e3a1604 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:41.648443 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a461a1-c73c-46e3-8aac-ac390e3a1604 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:41.671725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 310/619] 10.222.0.62 () {56 vars in 1135 bytes} [Fri Sep 23 13:43:41 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:41.677627 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a461a1-c73c-46e3-8aac-ac390e3a1604 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:41.708466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:41.751211 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b7921d2-9294-477f-9daf-3dfee1487d00 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:41.786099 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6726bc-e8ed-4513-9b93-3c2bb95dc515 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:41.787012 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6726bc-e8ed-4513-9b93-3c2bb95dc515 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:41.787918 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6726bc-e8ed-4513-9b93-3c2bb95dc515 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:41.788280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6726bc-e8ed-4513-9b93-3c2bb95dc515 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:41.805349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a461a1-c73c-46e3-8aac-ac390e3a1604 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:41.817719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6726bc-e8ed-4513-9b93-3c2bb95dc515 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:41.825086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-bc6726bc-e8ed-4513-9b93-3c2bb95dc515 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:41.825086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:41.825086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:41.825086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:41.825086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:41.825086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:41.825086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:41.825086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:41.825086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:41.825086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:41.825086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:41.825086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:41.825086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:41.825086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:41.825086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:41.825086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:41.826447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:41.826447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:41.826447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:41.826447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:41.826447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:41.826447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:41.826447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:41.826447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:41.826447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:41.826447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:41.826447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:41.826447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:41.826447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:41.826447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:41.826447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:41.826447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:41.827341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:41.827341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:41.827341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:41.827341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:41.827341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:41.827341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:41.827341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:41.827341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:41.827341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:41.827341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:41.827341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:41.827341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:41.831421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 310/620] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:43:41 2022] GET /identity/v3/roles/service => generated 85 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:41.837232 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6726bc-e8ed-4513-9b93-3c2bb95dc515 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:41.880680 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452355e5-592b-4652-8283-809d40023ad3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:41.881568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452355e5-592b-4652-8283-809d40023ad3 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:41.883658 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452355e5-592b-4652-8283-809d40023ad3 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:41.884185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452355e5-592b-4652-8283-809d40023ad3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:41.906850 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452355e5-592b-4652-8283-809d40023ad3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:41.908466 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452355e5-592b-4652-8283-809d40023ad3 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:41.922681 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:41.931806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-452355e5-592b-4652-8283-809d40023ad3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:41.931806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:41.931806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:41.931806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:41.931806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:41.931806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:41.931806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:41.931806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:41.931806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:41.931806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:41.931806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:41.931806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:41.931806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:41.931806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:41.931806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:41.931806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:41.932953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:41.932953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:41.932953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:41.932953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:41.932953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:41.932953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:41.932953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:41.932953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:41.932953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:41.932953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:41.932953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:41.932953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:41.932953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:41.932953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:41.932953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:41.932953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:41.932953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:41.934070 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:41.934070 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:41.934070 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:41.935813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 311/621] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:43:41.941713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452355e5-592b-4652-8283-809d40023ad3 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:42.013311 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6726bc-e8ed-4513-9b93-3c2bb95dc515 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:42.034568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:42.051534 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b026d26-ef26-4325-b7c8-8fa06f3d2939 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:42.052143 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b026d26-ef26-4325-b7c8-8fa06f3d2939 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:42.052420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b026d26-ef26-4325-b7c8-8fa06f3d2939 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:42.052741 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b026d26-ef26-4325-b7c8-8fa06f3d2939 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:42.053415 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b026d26-ef26-4325-b7c8-8fa06f3d2939 None admin] RBAC: Adding query filter params (name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:42.053715 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b026d26-ef26-4325-b7c8-8fa06f3d2939 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:42.055116 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b026d26-ef26-4325-b7c8-8fa06f3d2939 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:42.063719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 311/622] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:43:41 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:42.064571 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b026d26-ef26-4325-b7c8-8fa06f3d2939 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:42.087239 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452355e5-592b-4652-8283-809d40023ad3 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:42.135512 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5207bf7-c81c-4558-b9a2-fc739d25fa8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:42.136128 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5207bf7-c81c-4558-b9a2-fc739d25fa8e None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:42.136409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5207bf7-c81c-4558-b9a2-fc739d25fa8e None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:42.136994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5207bf7-c81c-4558-b9a2-fc739d25fa8e None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:42.138255 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5207bf7-c81c-4558-b9a2-fc739d25fa8e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:42.138766 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5207bf7-c81c-4558-b9a2-fc739d25fa8e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:42.140589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5207bf7-c81c-4558-b9a2-fc739d25fa8e None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:42.158414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 312/623] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:42.159185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5207bf7-c81c-4558-b9a2-fc739d25fa8e None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:42.176652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b026d26-ef26-4325-b7c8-8fa06f3d2939 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:42.224309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffeae07-84dc-456b-b083-62839f96cc7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:42.225403 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffeae07-84dc-456b-b083-62839f96cc7e None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:42.225755 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffeae07-84dc-456b-b083-62839f96cc7e None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:42.226134 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffeae07-84dc-456b-b083-62839f96cc7e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:42.238813 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ffeae07-84dc-456b-b083-62839f96cc7e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:42.241180 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:42.247224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-4ffeae07-84dc-456b-b083-62839f96cc7e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:42.247224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:42.247224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:42.247224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:42.247224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:42.247224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:42.247224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:42.247224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:42.247224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:42.247224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:42.247224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:42.247224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:42.247224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:42.247224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:42.247224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:42.247224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:42.248368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:42.248368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:42.248368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:42.248368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:42.248368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:42.248368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:42.248368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:42.248368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:42.248368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:42.248368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:42.248368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:42.248368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:42.248368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:42.248368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:42.248368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:42.248368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:42.249434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:42.249434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:42.249434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:42.249434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:42.249434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:42.249434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:42.249434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:42.249434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:42.249434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:42.249434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:42.249434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:42.249434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:42.254870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 312/624] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:43:42 2022] GET /identity/v3/roles/service => generated 85 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:42.255827 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffeae07-84dc-456b-b083-62839f96cc7e None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:42.295562 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5207bf7-c81c-4558-b9a2-fc739d25fa8e None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:42.310752 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:42.342152 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41b6f65-d5a1-42f4-82c7-eb363129f0c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:42.343045 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41b6f65-d5a1-42f4-82c7-eb363129f0c4 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:42.343969 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41b6f65-d5a1-42f4-82c7-eb363129f0c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:42.344418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41b6f65-d5a1-42f4-82c7-eb363129f0c4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:42.358123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41b6f65-d5a1-42f4-82c7-eb363129f0c4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:42.360441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41b6f65-d5a1-42f4-82c7-eb363129f0c4 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:42.380565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-b41b6f65-d5a1-42f4-82c7-eb363129f0c4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:42.380565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:42.380565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:42.380565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:42.380565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:42.380565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:42.380565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:42.380565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:42.380565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:42.380565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:42.380565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:42.380565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:42.380565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:42.380565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:42.380565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:42.380565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:42.382294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:42.382294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:42.382294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:42.382294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:42.382294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:42.382294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:42.382294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:42.382294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:42.382294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:42.382294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:42.382294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:42.382294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:42.382294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:42.382294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:42.382294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:42.382294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:42.382294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:42.383414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:42.383414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:42.383414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:42.385068 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffeae07-84dc-456b-b083-62839f96cc7e None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:42.385228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 313/625] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:42.386235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41b6f65-d5a1-42f4-82c7-eb363129f0c4 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:42.429864 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e60487-b797-4edb-8f02-bafb179968e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:42.430786 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e60487-b797-4edb-8f02-bafb179968e3 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:42.431281 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e60487-b797-4edb-8f02-bafb179968e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:42.431774 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e60487-b797-4edb-8f02-bafb179968e3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:42.439063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e60487-b797-4edb-8f02-bafb179968e3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:42.440728 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e60487-b797-4edb-8f02-bafb179968e3 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:42.454409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-e3e60487-b797-4edb-8f02-bafb179968e3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:42.454409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:42.454409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:42.454409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:42.454409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:42.454409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:42.454409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:42.454409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:42.454409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:42.454409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:42.454409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:42.454409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:42.454409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:43:42.454409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:43:42.454409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:43:42.455367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:43:42.455367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:42.455367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:42.455367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:42.455367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:42.455367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:42.455367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:42.455367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:42.455367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:42.455367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:42.455367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:42.455367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:42.455367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:42.455367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:42.455367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:42.455367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:42.455367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:42.456246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:43:42.456246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:43:42.456246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:43:42.456246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:43:42.456246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:43:42.456246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:43:42.456246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:42.456246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:42.458164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 313/626] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:42 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:42.459165 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e60487-b797-4edb-8f02-bafb179968e3 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:42.505400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41b6f65-d5a1-42f4-82c7-eb363129f0c4 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:42.517622 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:42.544416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaad9e46-6d7f-4e69-8592-0b99513abd05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:42.545179 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaad9e46-6d7f-4e69-8592-0b99513abd05 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:42.545565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaad9e46-6d7f-4e69-8592-0b99513abd05 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:42.545958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaad9e46-6d7f-4e69-8592-0b99513abd05 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:42.546772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaad9e46-6d7f-4e69-8592-0b99513abd05 None admin] RBAC: Adding query filter params (name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:42.547178 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaad9e46-6d7f-4e69-8592-0b99513abd05 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:42.548661 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaad9e46-6d7f-4e69-8592-0b99513abd05 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:42.563950 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 314/627] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:43:42 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:42.565031 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaad9e46-6d7f-4e69-8592-0b99513abd05 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:42.575899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:42.579250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e60487-b797-4edb-8f02-bafb179968e3 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:42.621610 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e57a5b-23d8-42fe-be2c-1779b69d1106 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:42.622567 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e57a5b-23d8-42fe-be2c-1779b69d1106 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:42.622991 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e57a5b-23d8-42fe-be2c-1779b69d1106 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:42.623407 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e57a5b-23d8-42fe-be2c-1779b69d1106 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:42.624382 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e57a5b-23d8-42fe-be2c-1779b69d1106 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:42.625218 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e57a5b-23d8-42fe-be2c-1779b69d1106 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:42.627044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e57a5b-23d8-42fe-be2c-1779b69d1106 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:42.641092 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 314/628] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:42.643071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e57a5b-23d8-42fe-be2c-1779b69d1106 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:42.685029 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaad9e46-6d7f-4e69-8592-0b99513abd05 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:42.723200 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba761a00-af0b-4edf-9efb-52d05f77e648 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:42.723811 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba761a00-af0b-4edf-9efb-52d05f77e648 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:42.724085 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba761a00-af0b-4edf-9efb-52d05f77e648 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:42.724346 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba761a00-af0b-4edf-9efb-52d05f77e648 None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:42.725568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba761a00-af0b-4edf-9efb-52d05f77e648 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:42.725860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba761a00-af0b-4edf-9efb-52d05f77e648 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:42.727299 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba761a00-af0b-4edf-9efb-52d05f77e648 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:42.738822 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 315/629] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:43:42 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:42.739621 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba761a00-af0b-4edf-9efb-52d05f77e648 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:42.771877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:42.773967 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e57a5b-23d8-42fe-be2c-1779b69d1106 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:42.814700 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303e6c75-5001-4aec-b3b1-f757549ccb80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:42.815326 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303e6c75-5001-4aec-b3b1-f757549ccb80 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:42.815601 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303e6c75-5001-4aec-b3b1-f757549ccb80 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:42.815868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303e6c75-5001-4aec-b3b1-f757549ccb80 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:42.826667 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303e6c75-5001-4aec-b3b1-f757549ccb80 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:42.828078 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303e6c75-5001-4aec-b3b1-f757549ccb80 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:42.842902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-303e6c75-5001-4aec-b3b1-f757549ccb80 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:42.842902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:42.842902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:42.842902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:42.842902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:42.842902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:42.842902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:42.842902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:42.842902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:42.842902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:42.842902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:42.842902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:42.842902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:42.842902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:42.842902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:42.842902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:42.843855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:42.843855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:42.843855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:42.843855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:42.843855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:42.843855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:42.843855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:42.843855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:42.843855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:42.843855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:42.843855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:42.843855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:42.843855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:42.843855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:42.843855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:42.843855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:42.843855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:42.844759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:42.844759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:42.844759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:42.844759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 315/630] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:42.845279 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73d2253-8d0a-48e4-a446-25de54234763 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:42.845463 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73d2253-8d0a-48e4-a446-25de54234763 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:42.845731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73d2253-8d0a-48e4-a446-25de54234763 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:42.846399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba761a00-af0b-4edf-9efb-52d05f77e648 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:42.850470 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 316/631] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:42 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:42.851419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73d2253-8d0a-48e4-a446-25de54234763 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:42.877087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:42.888033 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce495cf0-62a2-48ca-85e3-3b695dc55889 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:42.889070 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce495cf0-62a2-48ca-85e3-3b695dc55889 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:42.889070 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce495cf0-62a2-48ca-85e3-3b695dc55889 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:42.889498 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce495cf0-62a2-48ca-85e3-3b695dc55889 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:42.898815 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce495cf0-62a2-48ca-85e3-3b695dc55889 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:42.900266 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce495cf0-62a2-48ca-85e3-3b695dc55889 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:42.912487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-ce495cf0-62a2-48ca-85e3-3b695dc55889 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 23 13:43:42.912487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:42.912487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:42.912487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:42.912487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:42.912487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:42.912487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:42.912487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:42.912487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:42.912487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:42.912487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:42.912487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:42.912487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:43:42.912487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:43:42.912487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:43:42.912487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:43:42.913488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:42.913488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:42.913488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:43:42.913488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:42.913488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:43:42.913488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:42.913488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:42.913488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:42.913488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:42.913488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:42.913488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:42.913488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:42.913488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:42.913488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:42.913488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:42.913488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:42.913488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:42.913488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:42.914877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:42.914877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:42.914877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:43:42.914877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:43:42.914877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:43:42.914877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:43:42.914877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:43:42.914877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:43:42.914877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 23 13:43:42.914877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:42.918366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 316/632] 10.222.0.62 () {56 vars in 1093 bytes} [Fri Sep 23 13:43:42 2022] GET /identity/v3/users/nova => generated 82 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:42.919019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce495cf0-62a2-48ca-85e3-3b695dc55889 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:42.973127 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73d2253-8d0a-48e4-a446-25de54234763 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:43.008552 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacd7d73-45f8-44e8-8a1e-56244ca985ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:43.009447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacd7d73-45f8-44e8-8a1e-56244ca985ad None admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:43.009726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacd7d73-45f8-44e8-8a1e-56244ca985ad None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:43.010000 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacd7d73-45f8-44e8-8a1e-56244ca985ad None admin] PATH_INFO: `/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/4b9269509f1d454eab7a3008c2ba0039/roles/809612fc6d62442cbcc179b5c4123465` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:43.013832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:43.022313 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce495cf0-62a2-48ca-85e3-3b695dc55889 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:43.024438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacd7d73-45f8-44e8-8a1e-56244ca985ad None admin] RBAC: Authorizing `identity:create_grant(project_id=2a19fd4315ad4a6eb349dfe8cfa66c23, user_id=4b9269509f1d454eab7a3008c2ba0039, role_id=809612fc6d62442cbcc179b5c4123465)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:43.025865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bacd7d73-45f8-44e8-8a1e-56244ca985ad None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:43.055678 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b9f5fa-7498-4748-9c13-c766c9ac3865 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:43.056277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b9f5fa-7498-4748-9c13-c766c9ac3865 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:43.056556 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b9f5fa-7498-4748-9c13-c766c9ac3865 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:43.056825 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b9f5fa-7498-4748-9c13-c766c9ac3865 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:43.057501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b9f5fa-7498-4748-9c13-c766c9ac3865 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:43.057769 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b9f5fa-7498-4748-9c13-c766c9ac3865 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:43.059144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b9f5fa-7498-4748-9c13-c766c9ac3865 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:43.070808 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 317/633] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:43.071935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fff3132-2d30-4fdd-8ed4-cd12e6f06348 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:43.072216 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fff3132-2d30-4fdd-8ed4-cd12e6f06348 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:43.072490 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fff3132-2d30-4fdd-8ed4-cd12e6f06348 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:43.089538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-0fff3132-2d30-4fdd-8ed4-cd12e6f06348 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:43.090775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-0fff3132-2d30-4fdd-8ed4-cd12e6f06348 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:43.099392 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 317/634] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:43:42 2022] PUT /identity/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/4b9269509f1d454eab7a3008c2ba0039/roles/809612fc6d62442cbcc179b5c4123465 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:43:43.100500 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacd7d73-45f8-44e8-8a1e-56244ca985ad None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:43.178412 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 318/635] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:43.178672 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:43.187444 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc8869f-8889-499c-b2fc-8a2e274c9dfb None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:43.187745 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc8869f-8889-499c-b2fc-8a2e274c9dfb None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:43.187996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc8869f-8889-499c-b2fc-8a2e274c9dfb None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:43.194460 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 319/636] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:43 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:43.194460 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc8869f-8889-499c-b2fc-8a2e274c9dfb None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:43.232431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacd7d73-45f8-44e8-8a1e-56244ca985ad None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:43.279173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21cbb7b-d539-4c9c-83ae-de0aca3ce8db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:43.279796 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21cbb7b-d539-4c9c-83ae-de0aca3ce8db None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:43.280076 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21cbb7b-d539-4c9c-83ae-de0aca3ce8db None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:43.280348 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21cbb7b-d539-4c9c-83ae-de0aca3ce8db None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:43.281089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21cbb7b-d539-4c9c-83ae-de0aca3ce8db None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:43.281398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21cbb7b-d539-4c9c-83ae-de0aca3ce8db None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:43.282932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21cbb7b-d539-4c9c-83ae-de0aca3ce8db None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:43.308510 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 318/637] 10.222.0.62 () {56 vars in 1133 bytes} [Fri Sep 23 13:43:43 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:43.309633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccfacdf-2c27-4263-9d9c-c41876f19fe0 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:43.309919 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccfacdf-2c27-4263-9d9c-c41876f19fe0 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:43.310230 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccfacdf-2c27-4263-9d9c-c41876f19fe0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:43.332927 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-9ccfacdf-2c27-4263-9d9c-c41876f19fe0 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:43.335153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9ccfacdf-2c27-4263-9d9c-c41876f19fe0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:43.338062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:43.366232 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc8869f-8889-499c-b2fc-8a2e274c9dfb None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:43.415067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2164860-c10e-408d-bc09-8d1ed3db8fbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:43.416173 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2164860-c10e-408d-bc09-8d1ed3db8fbe None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:43.416652 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2164860-c10e-408d-bc09-8d1ed3db8fbe None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:43.417108 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2164860-c10e-408d-bc09-8d1ed3db8fbe None admin] PATH_INFO: `/v3/users/glance` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:43.418648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 319/638] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:43.423172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccfacdf-2c27-4263-9d9c-c41876f19fe0 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:43.431021 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2164860-c10e-408d-bc09-8d1ed3db8fbe None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:43.434648 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2164860-c10e-408d-bc09-8d1ed3db8fbe None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:43.450224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-a2164860-c10e-408d-bc09-8d1ed3db8fbe None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 23 13:43:43.450224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:43.450224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:43.450224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:43.450224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:43.450224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:43.450224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:43.450224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:43.450224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:43.450224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:43.450224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:43.450224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:43.450224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:43:43.450224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:43:43.450224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:43:43.450224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:43:43.451306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:43.451306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:43.451306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:43:43.451306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:43.451306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:43:43.451306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:43.451306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:43.451306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:43.451306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:43.451306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:43.451306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:43.451306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:43.451306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:43.451306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:43.451306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:43.451306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:43.451306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:43.451306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:43.452356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:43.452356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:43.452356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:43:43.452356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:43:43.452356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:43:43.452356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:43:43.452356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:43:43.452356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:43:43.452356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 23 13:43:43.452356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:43.455249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 320/639] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:43 2022] GET /identity/v3/users/glance => generated 84 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:43.462266 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2164860-c10e-408d-bc09-8d1ed3db8fbe None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:43.520603 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:43.581645 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccfacdf-2c27-4263-9d9c-c41876f19fe0 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:43.594233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2164860-c10e-408d-bc09-8d1ed3db8fbe None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:43.621320 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535cc3e5-eb84-4766-9f29-3beb7acbd897 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:43.622099 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535cc3e5-eb84-4766-9f29-3beb7acbd897 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:43.622548 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535cc3e5-eb84-4766-9f29-3beb7acbd897 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:43.622886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535cc3e5-eb84-4766-9f29-3beb7acbd897 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:43.640605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbca61b-7961-4809-902f-0c280ee3ae42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:43.641508 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbca61b-7961-4809-902f-0c280ee3ae42 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:43.641788 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbca61b-7961-4809-902f-0c280ee3ae42 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:43.642059 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbca61b-7961-4809-902f-0c280ee3ae42 None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:43.643005 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbca61b-7961-4809-902f-0c280ee3ae42 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:43.643617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbca61b-7961-4809-902f-0c280ee3ae42 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:43.645131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbca61b-7961-4809-902f-0c280ee3ae42 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:43.673804 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 321/640] 10.222.0.62 () {56 vars in 1137 bytes} [Fri Sep 23 13:43:43 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:43.674118 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:43.679087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbca61b-7961-4809-902f-0c280ee3ae42 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:43.720270 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535cc3e5-eb84-4766-9f29-3beb7acbd897 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:43.721750 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535cc3e5-eb84-4766-9f29-3beb7acbd897 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:43.778697 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:43.825312 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbca61b-7961-4809-902f-0c280ee3ae42 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:43.848266 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 320/641] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:43.849199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535cc3e5-eb84-4766-9f29-3beb7acbd897 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:43.870197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2939481-cd8e-4a4a-a887-04f26f8b15e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:43.870960 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2939481-cd8e-4a4a-a887-04f26f8b15e3 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:43.871237 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2939481-cd8e-4a4a-a887-04f26f8b15e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:43.871507 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2939481-cd8e-4a4a-a887-04f26f8b15e3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:43.882041 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2939481-cd8e-4a4a-a887-04f26f8b15e3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:43.887591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-b2939481-cd8e-4a4a-a887-04f26f8b15e3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:43.887591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:43.887591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:43.887591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:43.887591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:43.887591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:43.887591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:43.887591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:43.887591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:43.887591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:43.887591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:43.887591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:43.887591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:43.887591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:43.887591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:43.887591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:43.888736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:43.888736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:43.888736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:43.888736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:43.888736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:43.888736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:43.888736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:43.888736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:43.888736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:43.888736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:43.888736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:43.888736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:43.888736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:43.888736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:43.888736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:43.888736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:43.889605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:43.889605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:43.889605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:43.889605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:43.889605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:43.889605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:43.889605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:43.889605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:43.889605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:43.889605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:43.889605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:43.889605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:43.889605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 322/642] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:43:43 2022] GET /identity/v3/roles/service => generated 85 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:43.890357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2939481-cd8e-4a4a-a887-04f26f8b15e3 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:43.965571 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:43.977485 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535cc3e5-eb84-4766-9f29-3beb7acbd897 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:44.011003 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2939481-cd8e-4a4a-a887-04f26f8b15e3 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:44.017634 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6ab807-8ca4-41bf-aa2d-606c1e2df59f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:44.018367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6ab807-8ca4-41bf-aa2d-606c1e2df59f None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:44.018724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6ab807-8ca4-41bf-aa2d-606c1e2df59f None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:44.019062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6ab807-8ca4-41bf-aa2d-606c1e2df59f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:44.047324 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:44.055083 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f564d3ea-d304-4bff-90d1-21d3585f472a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:44.055722 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f564d3ea-d304-4bff-90d1-21d3585f472a None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:44.056001 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f564d3ea-d304-4bff-90d1-21d3585f472a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:44.056272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f564d3ea-d304-4bff-90d1-21d3585f472a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:44.071128 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f564d3ea-d304-4bff-90d1-21d3585f472a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:44.076105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-f564d3ea-d304-4bff-90d1-21d3585f472a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:44.076105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:44.076105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:44.076105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:44.076105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:44.076105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:44.076105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:44.076105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:44.076105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:44.076105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:44.076105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:44.076105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:44.076105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:44.076105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:44.076105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:44.076105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:44.077111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:44.077111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:44.077111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:44.077111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:44.077111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:44.077111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:44.077111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:44.077111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:44.077111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:44.077111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:44.077111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:44.077111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:44.077111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:44.077111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:44.077111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:44.077111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:44.077999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:44.077999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:44.077999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:44.077999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:44.077999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:44.077999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:44.077999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:44.077999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:44.077999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:44.077999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:44.077999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:44.077999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:44.077999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 323/643] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:43:43 2022] GET /identity/v3/roles/service => generated 85 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:44.077999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f564d3ea-d304-4bff-90d1-21d3585f472a None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:44.104456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6ab807-8ca4-41bf-aa2d-606c1e2df59f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:44.106319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6ab807-8ca4-41bf-aa2d-606c1e2df59f None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:44.213047 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f564d3ea-d304-4bff-90d1-21d3585f472a None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:44.225979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 321/644] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:44.227365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6ab807-8ca4-41bf-aa2d-606c1e2df59f None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:44.244481 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:44.268837 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483f0ce5-1601-44c1-8ebf-9ad693d5f5c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:44.269908 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483f0ce5-1601-44c1-8ebf-9ad693d5f5c2 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:44.270199 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483f0ce5-1601-44c1-8ebf-9ad693d5f5c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:44.270506 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483f0ce5-1601-44c1-8ebf-9ad693d5f5c2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:44.283293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483f0ce5-1601-44c1-8ebf-9ad693d5f5c2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:44.286137 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483f0ce5-1601-44c1-8ebf-9ad693d5f5c2 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:44.302784 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:44.304251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-483f0ce5-1601-44c1-8ebf-9ad693d5f5c2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:44.304251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:44.304251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:44.304251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:44.304251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:44.304251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:44.304251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:44.304251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:44.304251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:44.304251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:44.304251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:44.304251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:44.304251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:44.304251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:44.304251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:44.304251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:44.305373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:44.305373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:44.305373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:44.305373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:44.305373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:44.305373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:44.305373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:44.305373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:44.305373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:44.305373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:44.305373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:44.305373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:44.305373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:44.305373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:44.305373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:44.305373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:44.305373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:44.306673 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:44.306673 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:44.306673 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:44.307833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 324/645] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:44.308542 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483f0ce5-1601-44c1-8ebf-9ad693d5f5c2 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:44.368826 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6ab807-8ca4-41bf-aa2d-606c1e2df59f None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:44.411656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da3056b-76f0-44e2-bc8f-68d126ea8d0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:44.412401 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da3056b-76f0-44e2-bc8f-68d126ea8d0e None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:44.412899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da3056b-76f0-44e2-bc8f-68d126ea8d0e None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:44.413026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da3056b-76f0-44e2-bc8f-68d126ea8d0e None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:44.413732 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da3056b-76f0-44e2-bc8f-68d126ea8d0e None admin] RBAC: Adding query filter params (name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:44.414034 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da3056b-76f0-44e2-bc8f-68d126ea8d0e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:44.415464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da3056b-76f0-44e2-bc8f-68d126ea8d0e None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:44.419428 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483f0ce5-1601-44c1-8ebf-9ad693d5f5c2 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:44.428700 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 322/646] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:43:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:44.428700 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da3056b-76f0-44e2-bc8f-68d126ea8d0e None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:44.459757 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfb13d1-25fd-47eb-bf45-de819b11d55a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:44.460367 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfb13d1-25fd-47eb-bf45-de819b11d55a None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:44.460644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfb13d1-25fd-47eb-bf45-de819b11d55a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:44.460912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfb13d1-25fd-47eb-bf45-de819b11d55a None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:44.461575 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfb13d1-25fd-47eb-bf45-de819b11d55a None admin] RBAC: Adding query filter params (name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:44.461878 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfb13d1-25fd-47eb-bf45-de819b11d55a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:44.463273 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfb13d1-25fd-47eb-bf45-de819b11d55a None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:44.473644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 325/647] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:43:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:44.474614 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfb13d1-25fd-47eb-bf45-de819b11d55a None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:44.514769 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:44.547293 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da3056b-76f0-44e2-bc8f-68d126ea8d0e None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:44.562671 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:44.583381 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfb13d1-25fd-47eb-bf45-de819b11d55a None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:44.584820 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011caf25-c47b-4b6e-9c32-d89a74be8895 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:44.585446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011caf25-c47b-4b6e-9c32-d89a74be8895 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:44.585722 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011caf25-c47b-4b6e-9c32-d89a74be8895 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:44.585993 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011caf25-c47b-4b6e-9c32-d89a74be8895 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:44.596966 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011caf25-c47b-4b6e-9c32-d89a74be8895 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:44.599134 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011caf25-c47b-4b6e-9c32-d89a74be8895 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:44.613677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-011caf25-c47b-4b6e-9c32-d89a74be8895 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:44.613677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:44.613677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:44.613677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:44.613677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:44.613677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:44.613677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:44.613677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:44.613677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:44.613677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:44.613677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:44.613677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:44.613677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:44.613677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:44.613677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:44.613677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:44.614665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:44.614665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:44.614665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:44.614665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:44.614665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:44.614665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:44.614665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:44.614665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:44.614665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:44.614665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:44.614665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:44.614665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:44.614665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:44.614665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:44.614665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:44.614665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:44.614665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:44.616101 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:44.616101 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:44.616101 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:44.616101 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 323/648] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:44.616101 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011caf25-c47b-4b6e-9c32-d89a74be8895 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:44.624687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfc1b49-8e74-4ce2-90be-3ed781b43cfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:44.625699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfc1b49-8e74-4ce2-90be-3ed781b43cfb None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:44.625988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfc1b49-8e74-4ce2-90be-3ed781b43cfb None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:44.626317 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfc1b49-8e74-4ce2-90be-3ed781b43cfb None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:44.627034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfc1b49-8e74-4ce2-90be-3ed781b43cfb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:44.627721 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfc1b49-8e74-4ce2-90be-3ed781b43cfb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:44.629510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfc1b49-8e74-4ce2-90be-3ed781b43cfb None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:44.642041 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 326/649] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:44.642812 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfc1b49-8e74-4ce2-90be-3ed781b43cfb None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:44.726785 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011caf25-c47b-4b6e-9c32-d89a74be8895 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:44.741625 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:44.749094 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfc1b49-8e74-4ce2-90be-3ed781b43cfb None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:44.763783 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1299362-ec40-4bac-b5fe-ab0030805095 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:44.764493 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1299362-ec40-4bac-b5fe-ab0030805095 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:44.764778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1299362-ec40-4bac-b5fe-ab0030805095 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:44.765046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1299362-ec40-4bac-b5fe-ab0030805095 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:44.776419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1299362-ec40-4bac-b5fe-ab0030805095 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:44.778013 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1299362-ec40-4bac-b5fe-ab0030805095 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:44.786161 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b6ae50-3589-4c02-a4c5-61b997ab0d3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:44.786870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b6ae50-3589-4c02-a4c5-61b997ab0d3f None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:44.787332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b6ae50-3589-4c02-a4c5-61b997ab0d3f None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:44.787696 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b6ae50-3589-4c02-a4c5-61b997ab0d3f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:44.792895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-b1299362-ec40-4bac-b5fe-ab0030805095 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:44.792895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:44.792895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:44.792895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:44.792895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:44.792895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:44.792895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:44.792895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:44.792895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:44.792895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:44.792895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:44.792895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:44.792895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:44.792895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:44.792895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:44.792895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:44.793802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:44.793802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:44.793802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:44.793802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:44.793802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:44.793802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:44.793802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:44.793802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:44.793802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:44.793802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:44.793802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:44.793802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:44.793802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:44.793802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:44.793802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:44.793802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:44.793802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:44.794749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:44.794749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:44.794749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:44.794749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 324/650] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:44.795488 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1299362-ec40-4bac-b5fe-ab0030805095 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:44.803003 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b6ae50-3589-4c02-a4c5-61b997ab0d3f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:44.804990 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b6ae50-3589-4c02-a4c5-61b997ab0d3f None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:44.813550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:44.821111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-a2b6ae50-3589-4c02-a4c5-61b997ab0d3f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:44.821111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:44.821111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:44.821111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:44.821111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:44.821111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:44.821111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:44.821111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:44.821111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:44.821111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:44.821111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:44.821111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:44.821111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:44.821111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:44.821111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:44.821111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:44.822129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:44.822129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:44.822129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:44.822129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:44.822129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:44.822129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:44.822129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:44.822129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:44.822129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:44.822129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:44.822129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:44.822129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:44.822129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:44.822129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:44.822129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:44.822129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:44.822129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:44.827468 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:44.827468 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:44.827468 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:44.827468 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 327/651] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:44.827468 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b6ae50-3589-4c02-a4c5-61b997ab0d3f None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:44.907240 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1299362-ec40-4bac-b5fe-ab0030805095 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:44.930477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b6ae50-3589-4c02-a4c5-61b997ab0d3f None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:44.946192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8209f061-3baf-4edd-af67-9ad3ec7e7379 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:44.946845 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8209f061-3baf-4edd-af67-9ad3ec7e7379 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:44.947125 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8209f061-3baf-4edd-af67-9ad3ec7e7379 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:44.947397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8209f061-3baf-4edd-af67-9ad3ec7e7379 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:44.948042 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8209f061-3baf-4edd-af67-9ad3ec7e7379 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:44.948342 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8209f061-3baf-4edd-af67-9ad3ec7e7379 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:44.949694 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8209f061-3baf-4edd-af67-9ad3ec7e7379 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:44.966798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 325/652] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:44.966798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8209f061-3baf-4edd-af67-9ad3ec7e7379 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:44.969307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575aaf78-b349-45fc-83c6-a5223ac29e9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:44.969942 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575aaf78-b349-45fc-83c6-a5223ac29e9d None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:44.970221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575aaf78-b349-45fc-83c6-a5223ac29e9d None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:44.970510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575aaf78-b349-45fc-83c6-a5223ac29e9d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:44.978832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575aaf78-b349-45fc-83c6-a5223ac29e9d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:44.980505 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575aaf78-b349-45fc-83c6-a5223ac29e9d None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:44.993375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-575aaf78-b349-45fc-83c6-a5223ac29e9d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:44.993375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:44.993375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:44.993375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:44.993375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:44.993375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:44.993375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:44.993375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:44.993375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:44.993375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:44.993375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:44.993375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:44.993375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:43:44.993375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:43:44.993375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:43:44.994591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:43:44.994591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:44.994591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:44.994591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:44.994591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:44.994591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:44.994591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:44.994591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:44.994591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:44.994591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:44.994591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:44.994591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:44.994591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:44.994591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:44.994591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:44.994591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:44.994591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:44.995470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:43:44.995470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:43:44.995470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:43:44.995470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:43:44.995470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:43:44.995470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:43:44.995470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:44.995470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:44.995470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 328/653] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:44.995470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575aaf78-b349-45fc-83c6-a5223ac29e9d None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:45.006235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:45.054471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:45.078608 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8209f061-3baf-4edd-af67-9ad3ec7e7379 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:45.110488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575aaf78-b349-45fc-83c6-a5223ac29e9d None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:45.118471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c7dda1-846c-4a61-a57d-bf94f10af4bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:45.123584 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c7dda1-846c-4a61-a57d-bf94f10af4bf None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:45.123584 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c7dda1-846c-4a61-a57d-bf94f10af4bf None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:45.123584 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c7dda1-846c-4a61-a57d-bf94f10af4bf None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:45.123584 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c7dda1-846c-4a61-a57d-bf94f10af4bf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:45.123584 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c7dda1-846c-4a61-a57d-bf94f10af4bf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:45.123943 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c7dda1-846c-4a61-a57d-bf94f10af4bf None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:45.139255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 326/654] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:45.140734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c7dda1-846c-4a61-a57d-bf94f10af4bf None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:45.151331 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb4c7fd-4f87-4bee-939a-405c9f1a1ab7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:45.156801 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb4c7fd-4f87-4bee-939a-405c9f1a1ab7 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:45.156801 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb4c7fd-4f87-4bee-939a-405c9f1a1ab7 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:45.156801 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb4c7fd-4f87-4bee-939a-405c9f1a1ab7 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:45.156801 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb4c7fd-4f87-4bee-939a-405c9f1a1ab7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:45.156801 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb4c7fd-4f87-4bee-939a-405c9f1a1ab7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:45.156801 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb4c7fd-4f87-4bee-939a-405c9f1a1ab7 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:45.169982 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 329/655] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:45.175133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb4c7fd-4f87-4bee-939a-405c9f1a1ab7 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:45.248198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:45.270917 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c7dda1-846c-4a61-a57d-bf94f10af4bf None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:45.284668 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb4c7fd-4f87-4bee-939a-405c9f1a1ab7 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:45.307789 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108c9cb7-3b0c-489b-91a8-86d7312e9a61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:45.308415 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108c9cb7-3b0c-489b-91a8-86d7312e9a61 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:45.308692 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108c9cb7-3b0c-489b-91a8-86d7312e9a61 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:45.308959 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108c9cb7-3b0c-489b-91a8-86d7312e9a61 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:45.312003 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:45.316476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108c9cb7-3b0c-489b-91a8-86d7312e9a61 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:45.317893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-108c9cb7-3b0c-489b-91a8-86d7312e9a61 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:45.320579 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57f6c59-69e6-4ec5-9c1e-1938a80f1c69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:45.321197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57f6c59-69e6-4ec5-9c1e-1938a80f1c69 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:45.321476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57f6c59-69e6-4ec5-9c1e-1938a80f1c69 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:45.321744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57f6c59-69e6-4ec5-9c1e-1938a80f1c69 None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:45.322469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57f6c59-69e6-4ec5-9c1e-1938a80f1c69 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:45.322777 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57f6c59-69e6-4ec5-9c1e-1938a80f1c69 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:45.324461 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57f6c59-69e6-4ec5-9c1e-1938a80f1c69 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:45.329740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-108c9cb7-3b0c-489b-91a8-86d7312e9a61 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:45.329740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:45.329740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:45.329740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:45.329740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:45.329740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:45.329740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:45.329740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:45.329740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:45.329740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:45.329740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:45.329740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:45.329740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:43:45.329740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:43:45.329740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:43:45.330609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:43:45.330609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:45.330609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:45.330609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:45.330609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:45.330609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:45.330609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:45.330609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:45.330609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:45.330609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:45.330609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:45.330609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:45.330609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:45.330609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:45.330609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:45.330609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:45.330609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:45.337210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:43:45.337210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:43:45.337210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:43:45.337210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:43:45.337210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:43:45.337210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:43:45.337210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:45.337210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:45.337210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 327/656] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:45.337210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108c9cb7-3b0c-489b-91a8-86d7312e9a61 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:45.343758 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 330/657] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:43:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:45.349239 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57f6c59-69e6-4ec5-9c1e-1938a80f1c69 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:45.436388 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108c9cb7-3b0c-489b-91a8-86d7312e9a61 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:45.450179 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57f6c59-69e6-4ec5-9c1e-1938a80f1c69 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:45.469507 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5668c4e7-e5d6-4808-bdc6-eb7f01ec2719 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:45.470131 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5668c4e7-e5d6-4808-bdc6-eb7f01ec2719 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:45.470438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5668c4e7-e5d6-4808-bdc6-eb7f01ec2719 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:45.470707 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5668c4e7-e5d6-4808-bdc6-eb7f01ec2719 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:45.475205 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:45.478601 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5668c4e7-e5d6-4808-bdc6-eb7f01ec2719 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:45.480010 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5668c4e7-e5d6-4808-bdc6-eb7f01ec2719 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:45.484682 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5156569-58bf-4550-9e29-db13d501d2c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:45.488613 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5156569-58bf-4550-9e29-db13d501d2c2 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:45.488613 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5156569-58bf-4550-9e29-db13d501d2c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:45.488613 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5156569-58bf-4550-9e29-db13d501d2c2 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:45.492675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-5668c4e7-e5d6-4808-bdc6-eb7f01ec2719 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 23 13:43:45.492675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:45.492675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:45.492675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:45.492675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:45.492675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:45.492675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:45.492675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:45.492675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:45.492675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:45.492675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:45.492675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:45.492675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:43:45.492675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:43:45.492675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:43:45.492675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:43:45.493568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:45.493568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:45.493568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:43:45.493568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:45.493568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:43:45.493568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:45.493568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:45.493568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:45.493568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:45.493568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:45.493568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:45.493568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:45.493568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:45.493568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:45.493568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:45.493568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:45.493568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:45.493568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:45.494760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:45.494760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:45.494760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:43:45.494760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:43:45.494760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:43:45.494760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:43:45.494760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:43:45.494760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:43:45.494760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 23 13:43:45.494760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:45.494760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 328/658] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:45 2022] GET /identity/v3/users/cinder => generated 84 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:45.495875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5156569-58bf-4550-9e29-db13d501d2c2 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:45.495960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5668c4e7-e5d6-4808-bdc6-eb7f01ec2719 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:45.496035 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5156569-58bf-4550-9e29-db13d501d2c2 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:45.507659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-c5156569-58bf-4550-9e29-db13d501d2c2 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 23 13:43:45.507659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:45.507659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:45.507659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:45.507659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:45.507659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:45.507659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:45.507659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:45.507659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:45.507659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:45.507659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:45.507659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:45.507659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:43:45.507659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:43:45.507659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:43:45.507659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:43:45.508809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:45.508809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:45.508809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:43:45.508809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:45.508809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:43:45.508809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:45.508809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:45.508809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:45.508809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:45.508809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:45.508809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:45.508809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:45.508809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:45.508809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:45.508809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:45.508809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:45.508809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:45.508809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:45.510163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:45.510163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:45.510163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:43:45.510163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:43:45.510163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:43:45.510163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:43:45.510163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:43:45.510163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:43:45.510163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 23 13:43:45.510163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:45.512386 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 331/659] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:43:45 2022] GET /identity/v3/users/neutron => generated 85 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:45.513805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5156569-58bf-4550-9e29-db13d501d2c2 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:45.555009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:45.598364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5668c4e7-e5d6-4808-bdc6-eb7f01ec2719 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:45.611901 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5156569-58bf-4550-9e29-db13d501d2c2 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:45.633535 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39783b0e-48b6-49dc-8d32-b3c44808f738 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:45.634131 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39783b0e-48b6-49dc-8d32-b3c44808f738 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:45.634434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39783b0e-48b6-49dc-8d32-b3c44808f738 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:45.634703 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39783b0e-48b6-49dc-8d32-b3c44808f738 None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:45.635386 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39783b0e-48b6-49dc-8d32-b3c44808f738 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:45.635691 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39783b0e-48b6-49dc-8d32-b3c44808f738 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:45.637098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39783b0e-48b6-49dc-8d32-b3c44808f738 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:45.645478 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caaf4052-2f95-4444-b78a-0980a6034e69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:45.646113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caaf4052-2f95-4444-b78a-0980a6034e69 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:45.646628 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caaf4052-2f95-4444-b78a-0980a6034e69 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:45.646904 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caaf4052-2f95-4444-b78a-0980a6034e69 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:45.647616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caaf4052-2f95-4444-b78a-0980a6034e69 None admin] RBAC: Adding query filter params (role.id=809612fc6d62442cbcc179b5c4123465, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=f01ed94dbb384a178ad0a45a6f3d8c17) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:45.647920 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caaf4052-2f95-4444-b78a-0980a6034e69 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:45.648983 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 329/660] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:43:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:45.649356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caaf4052-2f95-4444-b78a-0980a6034e69 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:45.650139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4f2440-ca69-4f74-99cd-bbe439b5e19d None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:45.650449 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4f2440-ca69-4f74-99cd-bbe439b5e19d None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:45.650721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4f2440-ca69-4f74-99cd-bbe439b5e19d None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:45.652448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 330/661] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:45 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:45.652564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4f2440-ca69-4f74-99cd-bbe439b5e19d None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:45.655289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 332/662] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:43:45 2022] GET /identity/v3/role_assignments?user.id=f01ed94dbb384a178ad0a45a6f3d8c17&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=809612fc6d62442cbcc179b5c4123465 => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:45.656078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caaf4052-2f95-4444-b78a-0980a6034e69 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:45.714722 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:45.773073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caaf4052-2f95-4444-b78a-0980a6034e69 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:45.775456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4f2440-ca69-4f74-99cd-bbe439b5e19d None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:45.796565 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:45.813769 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907817b4-7b21-478c-9092-18dbce7ff333 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:45.814436 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907817b4-7b21-478c-9092-18dbce7ff333 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:45.814728 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907817b4-7b21-478c-9092-18dbce7ff333 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:45.814996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907817b4-7b21-478c-9092-18dbce7ff333 None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:45.815735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907817b4-7b21-478c-9092-18dbce7ff333 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:45.816097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907817b4-7b21-478c-9092-18dbce7ff333 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:45.817521 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907817b4-7b21-478c-9092-18dbce7ff333 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:45.818548 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b32f2b5-4df4-47ed-8d64-d188ba1eee49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:45.819166 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b32f2b5-4df4-47ed-8d64-d188ba1eee49 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:45.819443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b32f2b5-4df4-47ed-8d64-d188ba1eee49 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:45.819712 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b32f2b5-4df4-47ed-8d64-d188ba1eee49 None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:45.820451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b32f2b5-4df4-47ed-8d64-d188ba1eee49 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:45.820755 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b32f2b5-4df4-47ed-8d64-d188ba1eee49 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:45.822174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b32f2b5-4df4-47ed-8d64-d188ba1eee49 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:45.838905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 333/663] 10.222.0.62 () {56 vars in 1139 bytes} [Fri Sep 23 13:43:45 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:45.838905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907817b4-7b21-478c-9092-18dbce7ff333 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:45.843736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 331/664] 10.222.0.62 () {56 vars in 1137 bytes} [Fri Sep 23 13:43:45 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:45.844762 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca8c201-17f4-43db-b763-2351295e4583 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:45.845047 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca8c201-17f4-43db-b763-2351295e4583 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:45.845313 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca8c201-17f4-43db-b763-2351295e4583 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:45.866159 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-9ca8c201-17f4-43db-b763-2351295e4583 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:45.867237 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9ca8c201-17f4-43db-b763-2351295e4583 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:45.947571 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 332/665] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:45.948241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca8c201-17f4-43db-b763-2351295e4583 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:45.958436 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907817b4-7b21-478c-9092-18dbce7ff333 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:45.972358 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:46.006977 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b503d2d2-1f0d-4cc9-9f05-4a60ba8fcdc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:46.008383 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b503d2d2-1f0d-4cc9-9f05-4a60ba8fcdc1 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:46.008876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b503d2d2-1f0d-4cc9-9f05-4a60ba8fcdc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:46.009331 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b503d2d2-1f0d-4cc9-9f05-4a60ba8fcdc1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:46.010268 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b503d2d2-1f0d-4cc9-9f05-4a60ba8fcdc1 None admin] RBAC: Adding query filter params (role.id=809612fc6d62442cbcc179b5c4123465, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=acdf4a7bd608492390176dcc42ca1267) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:46.010795 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b503d2d2-1f0d-4cc9-9f05-4a60ba8fcdc1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:46.012394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b503d2d2-1f0d-4cc9-9f05-4a60ba8fcdc1 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:46.019743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 334/666] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:43:45 2022] GET /identity/v3/role_assignments?user.id=acdf4a7bd608492390176dcc42ca1267&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=809612fc6d62442cbcc179b5c4123465 => generated 608 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:46.020910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b503d2d2-1f0d-4cc9-9f05-4a60ba8fcdc1 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:46.064855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca8c201-17f4-43db-b763-2351295e4583 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:46.067496 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:46.110677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdfb9b4-a642-4430-90db-1f6a81e10202 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:46.112417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdfb9b4-a642-4430-90db-1f6a81e10202 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:46.112714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdfb9b4-a642-4430-90db-1f6a81e10202 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:46.112987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afdfb9b4-a642-4430-90db-1f6a81e10202 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:46.217985 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b503d2d2-1f0d-4cc9-9f05-4a60ba8fcdc1 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:46.241143 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afdfb9b4-a642-4430-90db-1f6a81e10202 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:46.242902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afdfb9b4-a642-4430-90db-1f6a81e10202 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:46.282100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9524eb9-0dd2-4231-ab77-36769677e1da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:46.282731 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9524eb9-0dd2-4231-ab77-36769677e1da None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:46.283003 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9524eb9-0dd2-4231-ab77-36769677e1da None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:46.283271 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9524eb9-0dd2-4231-ab77-36769677e1da None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:46.328327 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:46.362160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9524eb9-0dd2-4231-ab77-36769677e1da None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:46.363712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9524eb9-0dd2-4231-ab77-36769677e1da None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:46.370128 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 333/667] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:46.371276 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdfb9b4-a642-4430-90db-1f6a81e10202 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:46.375448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:46.480485 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 335/668] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 459 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:46.485300 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9524eb9-0dd2-4231-ab77-36769677e1da None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:46.498511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afdfb9b4-a642-4430-90db-1f6a81e10202 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:46.536532 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d992a4ef-0d95-4229-8397-28524f0cb3c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:46.538033 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d992a4ef-0d95-4229-8397-28524f0cb3c2 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:46.538439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d992a4ef-0d95-4229-8397-28524f0cb3c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:46.538804 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d992a4ef-0d95-4229-8397-28524f0cb3c2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:46.550229 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d992a4ef-0d95-4229-8397-28524f0cb3c2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:46.556350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-d992a4ef-0d95-4229-8397-28524f0cb3c2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:46.556350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:46.556350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:46.556350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:46.556350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:46.556350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:46.556350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:46.556350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:46.556350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:46.556350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:46.556350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:46.556350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:46.556350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:46.556350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:46.556350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:46.556350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:46.557301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:46.557301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:46.557301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:46.557301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:46.557301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:46.557301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:46.557301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:46.557301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:46.557301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:46.557301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:46.557301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:46.557301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:46.557301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:46.557301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:46.557301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:46.557301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:46.558160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:46.558160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:46.558160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:46.558160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:46.558160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:46.558160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:46.558160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:46.558160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:46.558160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:46.558160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:46.558160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:46.558160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:46.558160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 334/669] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:43:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:46.558880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d992a4ef-0d95-4229-8397-28524f0cb3c2 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:46.573920 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:46.617197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9524eb9-0dd2-4231-ab77-36769677e1da None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:46.641086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:46.670615 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61bc71b-a7af-4b6c-a101-ea030e541e42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:46.671251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61bc71b-a7af-4b6c-a101-ea030e541e42 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:46.671523 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61bc71b-a7af-4b6c-a101-ea030e541e42 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:46.671856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61bc71b-a7af-4b6c-a101-ea030e541e42 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:46.684743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61bc71b-a7af-4b6c-a101-ea030e541e42 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:46.686185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61bc71b-a7af-4b6c-a101-ea030e541e42 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:46.695486 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d992a4ef-0d95-4229-8397-28524f0cb3c2 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:46.700976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-c61bc71b-a7af-4b6c-a101-ea030e541e42 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:46.700976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:46.700976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:46.700976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:46.700976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:46.700976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:46.700976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:46.700976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:46.700976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:46.700976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:46.700976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:46.700976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:46.700976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:46.700976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:46.700976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:46.700976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:46.701931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:46.701931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:46.701931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:46.701931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:46.701931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:46.701931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:46.701931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:46.701931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:46.701931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:46.701931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:46.701931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:46.701931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:46.701931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:46.701931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:46.701931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:46.701931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:46.701931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:46.702850 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:46.702850 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:46.702850 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:46.706778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 336/670] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:46.707672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61bc71b-a7af-4b6c-a101-ea030e541e42 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:46.737954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea73865b-9e5c-4cbc-9677-0ad29cb0189c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:46.739117 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea73865b-9e5c-4cbc-9677-0ad29cb0189c None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:46.739617 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea73865b-9e5c-4cbc-9677-0ad29cb0189c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:46.740177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea73865b-9e5c-4cbc-9677-0ad29cb0189c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:46.752275 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea73865b-9e5c-4cbc-9677-0ad29cb0189c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:46.754124 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea73865b-9e5c-4cbc-9677-0ad29cb0189c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:46.771835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-ea73865b-9e5c-4cbc-9677-0ad29cb0189c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:46.771835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:46.771835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:46.771835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:46.771835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:46.771835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:46.771835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:46.771835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:46.771835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:46.771835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:46.771835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:46.771835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:46.771835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:46.771835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:46.771835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:46.771835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:46.772902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:46.772902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:46.772902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:46.772902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:46.772902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:46.772902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:46.772902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:46.772902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:46.772902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:46.772902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:46.772902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:46.772902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:46.772902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:46.772902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:46.772902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:46.772902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:46.772902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:46.773798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:46.773798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:46.773798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:46.775935 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 335/671] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:46.777489 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea73865b-9e5c-4cbc-9677-0ad29cb0189c None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:46.816275 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61bc71b-a7af-4b6c-a101-ea030e541e42 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:46.840258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:46.852127 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6552ca6c-c192-499e-a4c7-d58c78af7b74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:46.856229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6552ca6c-c192-499e-a4c7-d58c78af7b74 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:46.856229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6552ca6c-c192-499e-a4c7-d58c78af7b74 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:46.856229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6552ca6c-c192-499e-a4c7-d58c78af7b74 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:46.856229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6552ca6c-c192-499e-a4c7-d58c78af7b74 None admin] RBAC: Adding query filter params (name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:46.856229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6552ca6c-c192-499e-a4c7-d58c78af7b74 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:46.856229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6552ca6c-c192-499e-a4c7-d58c78af7b74 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:46.864519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 337/672] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:43:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:46.865637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6552ca6c-c192-499e-a4c7-d58c78af7b74 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:46.893892 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea73865b-9e5c-4cbc-9677-0ad29cb0189c None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:46.913832 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:46.934290 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98eb0532-0650-4d2e-bbc0-1655bcf4f1f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:46.935252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98eb0532-0650-4d2e-bbc0-1655bcf4f1f4 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:46.935729 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98eb0532-0650-4d2e-bbc0-1655bcf4f1f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:46.936301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98eb0532-0650-4d2e-bbc0-1655bcf4f1f4 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:46.937317 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98eb0532-0650-4d2e-bbc0-1655bcf4f1f4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:46.937876 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98eb0532-0650-4d2e-bbc0-1655bcf4f1f4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:46.939781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98eb0532-0650-4d2e-bbc0-1655bcf4f1f4 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:46.955092 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 336/673] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:46.956331 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98eb0532-0650-4d2e-bbc0-1655bcf4f1f4 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:46.973133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6552ca6c-c192-499e-a4c7-d58c78af7b74 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:47.009602 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3094ec6-7d4f-402f-ae39-c277bba0b100 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:47.010149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3094ec6-7d4f-402f-ae39-c277bba0b100 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:47.010455 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3094ec6-7d4f-402f-ae39-c277bba0b100 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:47.010725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3094ec6-7d4f-402f-ae39-c277bba0b100 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:47.011363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3094ec6-7d4f-402f-ae39-c277bba0b100 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:47.011659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3094ec6-7d4f-402f-ae39-c277bba0b100 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:47.013009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3094ec6-7d4f-402f-ae39-c277bba0b100 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:47.029852 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 338/674] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:47.030763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3094ec6-7d4f-402f-ae39-c277bba0b100 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:47.071985 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98eb0532-0650-4d2e-bbc0-1655bcf4f1f4 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:47.077967 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:47.111096 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544968b0-2789-4298-8076-6fea136a278c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:47.112098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544968b0-2789-4298-8076-6fea136a278c None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:47.112595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544968b0-2789-4298-8076-6fea136a278c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:47.113071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544968b0-2789-4298-8076-6fea136a278c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:47.126623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544968b0-2789-4298-8076-6fea136a278c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:47.128711 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544968b0-2789-4298-8076-6fea136a278c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:47.134922 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3094ec6-7d4f-402f-ae39-c277bba0b100 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:47.148312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-544968b0-2789-4298-8076-6fea136a278c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:47.148312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:47.148312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:47.148312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:47.148312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:47.148312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:47.148312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:47.148312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:47.148312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:47.148312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:47.148312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:47.148312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:47.148312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:47.148312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:47.148312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:47.148312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:47.151685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:47.151685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:47.151685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:47.151685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:47.151685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:47.151685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:47.151685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:47.151685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:47.151685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:47.151685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:47.151685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:47.151685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:47.151685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:47.151685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:47.151685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:47.151685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:47.151685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:47.152586 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:47.152586 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:47.152586 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:47.152586 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 337/675] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:47.154249 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544968b0-2789-4298-8076-6fea136a278c None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:47.171626 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86248072-e9f0-4682-b4ac-d1a8716ab965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:47.172254 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86248072-e9f0-4682-b4ac-d1a8716ab965 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:47.172615 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86248072-e9f0-4682-b4ac-d1a8716ab965 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:47.172800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86248072-e9f0-4682-b4ac-d1a8716ab965 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:47.180387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86248072-e9f0-4682-b4ac-d1a8716ab965 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:47.181809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86248072-e9f0-4682-b4ac-d1a8716ab965 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:47.188694 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:47.192999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-86248072-e9f0-4682-b4ac-d1a8716ab965 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:47.192999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:47.192999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:47.192999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:47.192999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:47.192999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:47.192999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:47.192999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:47.192999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:47.192999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:47.192999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:47.192999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:47.192999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:43:47.192999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:43:47.192999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:43:47.193882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:43:47.193882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:47.193882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:47.193882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:47.193882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:47.193882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:47.193882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:47.193882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:47.193882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:47.193882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:47.193882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:47.193882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:47.193882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:47.193882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:47.193882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:47.193882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:47.193882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:47.195160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:43:47.195160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:43:47.195160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:43:47.195160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:43:47.195160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:43:47.195160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:43:47.195160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:47.195160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:47.198485 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 339/676] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:47.199303 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86248072-e9f0-4682-b4ac-d1a8716ab965 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:47.278853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544968b0-2789-4298-8076-6fea136a278c None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:47.316374 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5519cc-d99a-4b35-9c76-6a9486cfb625 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:47.317417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5519cc-d99a-4b35-9c76-6a9486cfb625 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:47.318587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5519cc-d99a-4b35-9c76-6a9486cfb625 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:47.319062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5519cc-d99a-4b35-9c76-6a9486cfb625 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:47.326637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5519cc-d99a-4b35-9c76-6a9486cfb625 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:47.327486 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86248072-e9f0-4682-b4ac-d1a8716ab965 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:47.328557 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5519cc-d99a-4b35-9c76-6a9486cfb625 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:47.335590 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:47.342863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-ef5519cc-d99a-4b35-9c76-6a9486cfb625 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:47.342863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:47.342863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:47.342863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:47.342863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:47.342863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:47.342863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:47.342863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:47.342863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:47.342863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:47.342863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:47.342863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:47.342863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:43:47.342863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:43:47.342863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:43:47.343749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:43:47.343749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:47.343749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:47.343749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:47.343749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:47.343749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:47.343749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:47.343749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:47.343749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:47.343749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:47.343749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:47.343749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:47.343749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:47.343749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:47.343749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:47.343749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:47.343749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:47.344626 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:43:47.344626 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:43:47.344626 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:43:47.344626 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:43:47.344626 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:43:47.344626 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:43:47.344626 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:47.344626 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:47.347132 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 338/677] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:47.348259 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5519cc-d99a-4b35-9c76-6a9486cfb625 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:47.371470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5164b5c7-c13c-4d40-afc8-8a2aea169a4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:47.372094 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5164b5c7-c13c-4d40-afc8-8a2aea169a4a None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:47.372497 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5164b5c7-c13c-4d40-afc8-8a2aea169a4a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:47.372772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5164b5c7-c13c-4d40-afc8-8a2aea169a4a None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:47.373408 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5164b5c7-c13c-4d40-afc8-8a2aea169a4a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:47.373711 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5164b5c7-c13c-4d40-afc8-8a2aea169a4a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:47.375424 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5164b5c7-c13c-4d40-afc8-8a2aea169a4a None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:47.391222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 340/678] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:47.392952 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5164b5c7-c13c-4d40-afc8-8a2aea169a4a None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:47.454773 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:47.468018 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5519cc-d99a-4b35-9c76-6a9486cfb625 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:47.509134 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a5e43a-2d8c-4b76-b6ba-dba5dcc526a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:47.509748 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a5e43a-2d8c-4b76-b6ba-dba5dcc526a8 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:47.510144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a5e43a-2d8c-4b76-b6ba-dba5dcc526a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:47.510440 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a5e43a-2d8c-4b76-b6ba-dba5dcc526a8 None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:47.511137 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a5e43a-2d8c-4b76-b6ba-dba5dcc526a8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:47.511444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a5e43a-2d8c-4b76-b6ba-dba5dcc526a8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:47.513137 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a5e43a-2d8c-4b76-b6ba-dba5dcc526a8 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:47.533005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 339/679] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:43:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:47.533005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a5e43a-2d8c-4b76-b6ba-dba5dcc526a8 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:47.538283 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5164b5c7-c13c-4d40-afc8-8a2aea169a4a None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:47.590225 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6bb757-08e8-4130-8b7b-3b2aec551e9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:47.590912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6bb757-08e8-4130-8b7b-3b2aec551e9b None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:47.591192 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6bb757-08e8-4130-8b7b-3b2aec551e9b None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:47.591510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6bb757-08e8-4130-8b7b-3b2aec551e9b None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:47.592209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6bb757-08e8-4130-8b7b-3b2aec551e9b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:47.592512 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6bb757-08e8-4130-8b7b-3b2aec551e9b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:47.593941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6bb757-08e8-4130-8b7b-3b2aec551e9b None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:47.612115 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 341/680] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:43:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:47.612115 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6bb757-08e8-4130-8b7b-3b2aec551e9b None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:47.641813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:47.667849 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a5e43a-2d8c-4b76-b6ba-dba5dcc526a8 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:47.707419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d405e3-ec85-4d81-8f8d-31c955571965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:47.709051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d405e3-ec85-4d81-8f8d-31c955571965 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:47.710118 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d405e3-ec85-4d81-8f8d-31c955571965 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:47.710614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d405e3-ec85-4d81-8f8d-31c955571965 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:47.719723 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6bb757-08e8-4130-8b7b-3b2aec551e9b None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:47.720107 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d405e3-ec85-4d81-8f8d-31c955571965 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:47.721720 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d405e3-ec85-4d81-8f8d-31c955571965 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:47.736389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-f6d405e3-ec85-4d81-8f8d-31c955571965 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 23 13:43:47.736389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:47.736389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:47.736389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:47.736389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:47.736389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:47.736389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:47.736389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:47.736389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:47.736389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:47.736389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:47.736389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:47.736389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:43:47.736389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:43:47.736389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:43:47.736389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:43:47.737430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:47.737430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:47.737430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:43:47.737430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:47.737430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:43:47.737430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:47.737430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:47.737430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:47.737430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:47.737430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:47.737430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:47.737430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:47.737430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:47.737430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:47.737430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:47.737430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:47.737430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:47.737430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:47.738428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:47.738428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:47.738428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:43:47.738428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:43:47.738428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:43:47.738428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:43:47.738428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:43:47.738428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:43:47.738428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 23 13:43:47.738428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:47.741152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 340/681] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:43:47 2022] GET /identity/v3/users/swift => generated 83 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:47.742131 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d405e3-ec85-4d81-8f8d-31c955571965 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:47.757850 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:47.764727 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb201253-4d9f-4a3a-8558-49cbb6fc3b7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:47.765694 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb201253-4d9f-4a3a-8558-49cbb6fc3b7e None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:47.766136 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb201253-4d9f-4a3a-8558-49cbb6fc3b7e None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:47.766869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb201253-4d9f-4a3a-8558-49cbb6fc3b7e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:47.767751 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb201253-4d9f-4a3a-8558-49cbb6fc3b7e None admin] RBAC: Adding query filter params (role.id=809612fc6d62442cbcc179b5c4123465, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=580a35627c234a25aa04f5d376177ca5) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:47.768208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb201253-4d9f-4a3a-8558-49cbb6fc3b7e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:47.770241 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb201253-4d9f-4a3a-8558-49cbb6fc3b7e None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:47.786492 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 342/682] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:43:47 2022] GET /identity/v3/role_assignments?user.id=580a35627c234a25aa04f5d376177ca5&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=809612fc6d62442cbcc179b5c4123465 => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:47.786492 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb201253-4d9f-4a3a-8558-49cbb6fc3b7e None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:47.940142 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:47.957222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb201253-4d9f-4a3a-8558-49cbb6fc3b7e None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:47.962871 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d405e3-ec85-4d81-8f8d-31c955571965 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:48.007666 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f000fdc-1c64-4aae-9b13-8cefad9b0b7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:48.008295 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f000fdc-1c64-4aae-9b13-8cefad9b0b7d None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:48.008577 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f000fdc-1c64-4aae-9b13-8cefad9b0b7d None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:48.008848 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f000fdc-1c64-4aae-9b13-8cefad9b0b7d None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:48.009592 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f000fdc-1c64-4aae-9b13-8cefad9b0b7d None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:48.010418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f000fdc-1c64-4aae-9b13-8cefad9b0b7d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:48.012049 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f000fdc-1c64-4aae-9b13-8cefad9b0b7d None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:48.030310 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c0c75a-e05b-4430-ae5e-f87d3495bdbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:48.031565 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c0c75a-e05b-4430-ae5e-f87d3495bdbf None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:48.032079 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c0c75a-e05b-4430-ae5e-f87d3495bdbf None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:48.032564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c0c75a-e05b-4430-ae5e-f87d3495bdbf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:48.033525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 343/683] 10.222.0.62 () {56 vars in 1135 bytes} [Fri Sep 23 13:43:47 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:48.033648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c0c75a-e05b-4430-ae5e-f87d3495bdbf None admin] RBAC: Adding query filter params (role.id=809612fc6d62442cbcc179b5c4123465, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=fae68ca0754a4b0b88a57f39efa2fb45) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:48.034089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c0c75a-e05b-4430-ae5e-f87d3495bdbf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:48.036197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76858cab-8ab9-4e32-8750-3a74dc0c95b0 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:48.036197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76858cab-8ab9-4e32-8750-3a74dc0c95b0 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:48.036197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76858cab-8ab9-4e32-8750-3a74dc0c95b0 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:48.037045 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c0c75a-e05b-4430-ae5e-f87d3495bdbf None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:48.038479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 344/684] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:48.042363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76858cab-8ab9-4e32-8750-3a74dc0c95b0 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:48.051575 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 341/685] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:43:47 2022] GET /identity/v3/role_assignments?user.id=fae68ca0754a4b0b88a57f39efa2fb45&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=809612fc6d62442cbcc179b5c4123465 => generated 608 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:48.055009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17e363c-643a-499d-a31a-529206040295 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:48.055300 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17e363c-643a-499d-a31a-529206040295 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:48.055574 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17e363c-643a-499d-a31a-529206040295 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:48.075653 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-c17e363c-643a-499d-a31a-529206040295 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:48.077254 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-c17e363c-643a-499d-a31a-529206040295 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:48.175779 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 342/686] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:48.219757 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76858cab-8ab9-4e32-8750-3a74dc0c95b0 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:48.273426 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d433d82-efca-4811-94b3-a5294cbb7282 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:48.274481 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d433d82-efca-4811-94b3-a5294cbb7282 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:48.274961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d433d82-efca-4811-94b3-a5294cbb7282 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:48.275424 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d433d82-efca-4811-94b3-a5294cbb7282 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:48.303883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:48.365516 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d433d82-efca-4811-94b3-a5294cbb7282 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:48.367192 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d433d82-efca-4811-94b3-a5294cbb7282 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:48.478294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 345/687] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 436 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 23 13:43:48.483928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17e363c-643a-499d-a31a-529206040295 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:48.486533 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:48.537874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d433d82-efca-4811-94b3-a5294cbb7282 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:48.599730 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17e363c-643a-499d-a31a-529206040295 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:48.623273 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:48.643158 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b24925-9c37-4d4e-bfa8-1c4acc385d37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:48.644235 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b24925-9c37-4d4e-bfa8-1c4acc385d37 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:48.644709 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b24925-9c37-4d4e-bfa8-1c4acc385d37 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:48.645170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b24925-9c37-4d4e-bfa8-1c4acc385d37 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:48.658124 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b24925-9c37-4d4e-bfa8-1c4acc385d37 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:48.660383 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b24925-9c37-4d4e-bfa8-1c4acc385d37 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:48.678465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-c1b24925-9c37-4d4e-bfa8-1c4acc385d37 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:48.678465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:48.678465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:48.678465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:48.678465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:48.678465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:48.678465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:48.678465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:48.678465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:48.678465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:48.678465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:48.678465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:48.678465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:48.678465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:48.678465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:48.678465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:48.679438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:48.679438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:48.679438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:48.679438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:48.679438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:48.679438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:48.679438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:48.679438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:48.679438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:48.679438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:48.679438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:48.679438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:48.679438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:48.679438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:48.679438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:48.679438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:48.679438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:48.680297 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:48.680297 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:48.680297 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:48.682960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 343/688] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:48.684356 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54807482-de52-460b-ac6d-4cc98365cd93 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:48.684831 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54807482-de52-460b-ac6d-4cc98365cd93 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:48.685292 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54807482-de52-460b-ac6d-4cc98365cd93 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:48.687143 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 344/689] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:48.688070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54807482-de52-460b-ac6d-4cc98365cd93 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:48.694412 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d433d82-efca-4811-94b3-a5294cbb7282 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:48.738933 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a90595-5f41-4f21-8a4c-2835ff8e0bc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:48.740148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a90595-5f41-4f21-8a4c-2835ff8e0bc0 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:48.740399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a90595-5f41-4f21-8a4c-2835ff8e0bc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:48.740730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a90595-5f41-4f21-8a4c-2835ff8e0bc0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:48.755362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a90595-5f41-4f21-8a4c-2835ff8e0bc0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:48.757146 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:48.762247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-51a90595-5f41-4f21-8a4c-2835ff8e0bc0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:43:48.762247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:48.762247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:48.762247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:48.762247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:48.762247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:48.762247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:48.762247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:48.762247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:48.762247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:48.762247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:48.762247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:48.762247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:48.762247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:48.762247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:48.762247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:48.763307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:48.763307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:48.763307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:48.763307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:48.763307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:48.763307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:48.763307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:48.763307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:48.763307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:48.763307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:48.763307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:48.763307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:48.763307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:48.763307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:48.763307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:48.763307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:48.764156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:48.764156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:48.764156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:48.764156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:48.764156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:48.764156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:48.764156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:48.764156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:48.764156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:48.764156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:48.764156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:43:48.764156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:48.764156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 346/690] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:43:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:48.765296 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f335a324-c530-49de-b0ab-ea697e661326 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:48.765296 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f335a324-c530-49de-b0ab-ea697e661326 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:48.765519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f335a324-c530-49de-b0ab-ea697e661326 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:48.789241 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-f335a324-c530-49de-b0ab-ea697e661326 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:48.791544 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f335a324-c530-49de-b0ab-ea697e661326 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:48.817389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54807482-de52-460b-ac6d-4cc98365cd93 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:48.857091 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361bce1c-d543-41a1-853d-6d018068a0b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:48.857643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361bce1c-d543-41a1-853d-6d018068a0b2 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:48.857890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361bce1c-d543-41a1-853d-6d018068a0b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:48.858159 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361bce1c-d543-41a1-853d-6d018068a0b2 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:48.858830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361bce1c-d543-41a1-853d-6d018068a0b2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:48.859131 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361bce1c-d543-41a1-853d-6d018068a0b2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:48.860519 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361bce1c-d543-41a1-853d-6d018068a0b2 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:48.865381 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 347/691] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:48.866549 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f335a324-c530-49de-b0ab-ea697e661326 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:48.874663 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 345/692] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:48.880084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361bce1c-d543-41a1-853d-6d018068a0b2 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:48.939198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:49.017935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f335a324-c530-49de-b0ab-ea697e661326 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:49.058221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440afa7b-69a3-40a2-b173-86675e9dae9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:49.059554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440afa7b-69a3-40a2-b173-86675e9dae9a None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:49.060029 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440afa7b-69a3-40a2-b173-86675e9dae9a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:49.060493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440afa7b-69a3-40a2-b173-86675e9dae9a None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:49.061533 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440afa7b-69a3-40a2-b173-86675e9dae9a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:49.062025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440afa7b-69a3-40a2-b173-86675e9dae9a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:49.063690 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-440afa7b-69a3-40a2-b173-86675e9dae9a None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:49.074535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 348/693] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:49.086403 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440afa7b-69a3-40a2-b173-86675e9dae9a None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:49.104311 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361bce1c-d543-41a1-853d-6d018068a0b2 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:49.137086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:49.173835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826c9097-638d-461d-b860-1aa0b50d02d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:49.175601 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826c9097-638d-461d-b860-1aa0b50d02d8 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:49.175886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826c9097-638d-461d-b860-1aa0b50d02d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:49.176156 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826c9097-638d-461d-b860-1aa0b50d02d8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:49.194026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826c9097-638d-461d-b860-1aa0b50d02d8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:49.195559 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826c9097-638d-461d-b860-1aa0b50d02d8 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:49.218887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-826c9097-638d-461d-b860-1aa0b50d02d8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:49.218887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:49.218887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:49.218887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:49.218887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:49.218887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:49.218887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:49.218887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:49.218887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:49.218887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:49.218887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:49.218887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:49.218887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:43:49.218887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:43:49.218887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:43:49.219942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:43:49.219942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:49.219942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:49.219942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:49.219942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:49.219942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:49.219942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:49.219942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:49.219942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:49.219942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:49.219942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:49.219942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:49.219942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:49.219942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:49.219942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:49.219942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:49.219942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:49.221051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:43:49.221051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:43:49.221051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:43:49.221051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:43:49.221051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:43:49.221051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:43:49.221051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:49.221051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:49.222720 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 346/694] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 343 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:49.223912 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826c9097-638d-461d-b860-1aa0b50d02d8 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:49.263259 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:49.266237 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440afa7b-69a3-40a2-b173-86675e9dae9a None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:49.305541 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce849d5-4d52-4f82-b240-cd2ae0346036 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:49.306324 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce849d5-4d52-4f82-b240-cd2ae0346036 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:49.306724 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce849d5-4d52-4f82-b240-cd2ae0346036 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:49.307114 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce849d5-4d52-4f82-b240-cd2ae0346036 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:49.322564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce849d5-4d52-4f82-b240-cd2ae0346036 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:49.324699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce849d5-4d52-4f82-b240-cd2ae0346036 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:49.345441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-cce849d5-4d52-4f82-b240-cd2ae0346036 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:49.345441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:49.345441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:49.345441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:49.345441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:49.345441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:49.345441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:49.345441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:49.345441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:49.345441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:49.345441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:49.345441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:49.345441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:49.345441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:49.345441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:49.345441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:49.346634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:49.346634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:49.346634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:49.346634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:49.346634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:49.346634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:49.346634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:49.346634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:49.346634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:49.346634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:49.346634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:49.346634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:49.346634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:49.346634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:49.346634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:49.346634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:49.346634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:49.347733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:49.347733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:49.347733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:49.354683 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 349/695] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 265 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:49.356038 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce849d5-4d52-4f82-b240-cd2ae0346036 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:49.358655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826c9097-638d-461d-b860-1aa0b50d02d8 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:49.397212 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912feacb-6e9a-4041-a91e-7d8a9635750a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:49.398426 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912feacb-6e9a-4041-a91e-7d8a9635750a None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:49.398849 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912feacb-6e9a-4041-a91e-7d8a9635750a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:49.399333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912feacb-6e9a-4041-a91e-7d8a9635750a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:49.410308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912feacb-6e9a-4041-a91e-7d8a9635750a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:49.412821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912feacb-6e9a-4041-a91e-7d8a9635750a None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:49.429452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-912feacb-6e9a-4041-a91e-7d8a9635750a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:49.429452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:49.429452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:49.429452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:49.429452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:49.429452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:49.429452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:49.429452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:49.429452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:49.429452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:49.429452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:49.429452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:49.429452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:49.429452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:49.429452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:49.429452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:49.430465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:49.430465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:49.430465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:49.430465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:49.430465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:49.430465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:49.430465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:49.430465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:49.430465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:49.430465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:49.430465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:49.430465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:49.430465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:49.430465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:49.430465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:49.430465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:49.430465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:49.431339 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:49.431339 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:49.431339 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:49.431339 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 347/696] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:49.432024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912feacb-6e9a-4041-a91e-7d8a9635750a None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:49.474719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:49.491389 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce849d5-4d52-4f82-b240-cd2ae0346036 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:49.536009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6fc1d7-7a0b-4e0e-82e9-978ad6cf72e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:49.536761 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6fc1d7-7a0b-4e0e-82e9-978ad6cf72e6 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:49.537044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6fc1d7-7a0b-4e0e-82e9-978ad6cf72e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:49.537315 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6fc1d7-7a0b-4e0e-82e9-978ad6cf72e6 None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:49.538006 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6fc1d7-7a0b-4e0e-82e9-978ad6cf72e6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:49.538310 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6fc1d7-7a0b-4e0e-82e9-978ad6cf72e6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:49.539770 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6fc1d7-7a0b-4e0e-82e9-978ad6cf72e6 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:49.553823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 350/697] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:43:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:49.554766 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6fc1d7-7a0b-4e0e-82e9-978ad6cf72e6 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:49.564496 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912feacb-6e9a-4041-a91e-7d8a9635750a None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:49.570125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:49.605516 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c338081-0795-4460-8003-ff866443db78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:49.606455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c338081-0795-4460-8003-ff866443db78 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:49.606926 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c338081-0795-4460-8003-ff866443db78 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:49.607526 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c338081-0795-4460-8003-ff866443db78 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:49.608614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c338081-0795-4460-8003-ff866443db78 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:49.609107 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c338081-0795-4460-8003-ff866443db78 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:49.610660 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c338081-0795-4460-8003-ff866443db78 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:49.626174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 348/698] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:49.627336 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c338081-0795-4460-8003-ff866443db78 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:49.665133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6fc1d7-7a0b-4e0e-82e9-978ad6cf72e6 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:49.701525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b1dde4-83f8-4c8b-89fc-f57b4757ad5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:49.702497 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b1dde4-83f8-4c8b-89fc-f57b4757ad5a None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:49.702970 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b1dde4-83f8-4c8b-89fc-f57b4757ad5a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:49.703448 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b1dde4-83f8-4c8b-89fc-f57b4757ad5a None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:49.704286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b1dde4-83f8-4c8b-89fc-f57b4757ad5a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:49.704682 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b1dde4-83f8-4c8b-89fc-f57b4757ad5a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:49.706203 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b1dde4-83f8-4c8b-89fc-f57b4757ad5a None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:49.722582 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 351/699] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:49.722945 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:49.723697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b1dde4-83f8-4c8b-89fc-f57b4757ad5a None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:49.742066 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c338081-0795-4460-8003-ff866443db78 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:49.789020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fd48e7-c8d3-422c-9dfc-e18860760e4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:49.789901 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fd48e7-c8d3-422c-9dfc-e18860760e4b None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:49.790394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fd48e7-c8d3-422c-9dfc-e18860760e4b None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:49.790805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83fd48e7-c8d3-422c-9dfc-e18860760e4b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:49.792359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83fd48e7-c8d3-422c-9dfc-e18860760e4b None admin] RBAC: Adding query filter params (role.id=809612fc6d62442cbcc179b5c4123465, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=4b9269509f1d454eab7a3008c2ba0039) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:49.792959 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83fd48e7-c8d3-422c-9dfc-e18860760e4b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:49.794655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83fd48e7-c8d3-422c-9dfc-e18860760e4b None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:49.801101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:49.804295 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 349/700] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:43:49 2022] GET /identity/v3/role_assignments?user.id=4b9269509f1d454eab7a3008c2ba0039&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=809612fc6d62442cbcc179b5c4123465 => generated 608 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:49.805341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fd48e7-c8d3-422c-9dfc-e18860760e4b None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:49.832329 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b1dde4-83f8-4c8b-89fc-f57b4757ad5a None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:49.899925 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbd55b1-78aa-4bbf-b116-983634064cef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:49.900571 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbd55b1-78aa-4bbf-b116-983634064cef None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:49.900848 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbd55b1-78aa-4bbf-b116-983634064cef None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:49.901114 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbd55b1-78aa-4bbf-b116-983634064cef None admin] PATH_INFO: `/v3/users/nova` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:49.914874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbd55b1-78aa-4bbf-b116-983634064cef None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:49.916376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbd55b1-78aa-4bbf-b116-983634064cef None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:49.933749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-5dbd55b1-78aa-4bbf-b116-983634064cef None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 23 13:43:49.933749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:49.933749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:49.933749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:49.933749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:49.933749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:49.933749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:49.933749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:49.933749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:49.933749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:49.933749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:49.933749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:49.933749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:43:49.933749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:43:49.933749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:43:49.933749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:43:49.934814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:49.934814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:49.934814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:43:49.934814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:49.934814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:43:49.934814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:49.934814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:49.934814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:49.934814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:49.934814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:49.934814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:49.934814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:49.934814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:49.934814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:49.934814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:49.934814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:49.934814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:49.934814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:49.936385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:49.936385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:49.936385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:43:49.936385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:43:49.936385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:43:49.936385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:43:49.936385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:43:49.936385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:43:49.936385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 23 13:43:49.936385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:49.936385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 352/701] 10.222.0.62 () {56 vars in 1093 bytes} [Fri Sep 23 13:43:49 2022] GET /identity/v3/users/nova => generated 82 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:49.944078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbd55b1-78aa-4bbf-b116-983634064cef None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:50.015288 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83fd48e7-c8d3-422c-9dfc-e18860760e4b None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:50.070942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0cd721-df7f-432c-960f-d2da6f264204 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:50.071558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0cd721-df7f-432c-960f-d2da6f264204 None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:50.071839 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0cd721-df7f-432c-960f-d2da6f264204 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:50.072112 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0cd721-df7f-432c-960f-d2da6f264204 None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:50.072765 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0cd721-df7f-432c-960f-d2da6f264204 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:50.074191 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0cd721-df7f-432c-960f-d2da6f264204 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:50.106573 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbd55b1-78aa-4bbf-b116-983634064cef None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:50.141926 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:50.149596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2e996a-1721-4dee-b761-49fd74cc8ea4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:50.150307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2e996a-1721-4dee-b761-49fd74cc8ea4 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:50.150760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2e996a-1721-4dee-b761-49fd74cc8ea4 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:50.151090 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2e996a-1721-4dee-b761-49fd74cc8ea4 None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:50.151886 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2e996a-1721-4dee-b761-49fd74cc8ea4 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:50.152296 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2e996a-1721-4dee-b761-49fd74cc8ea4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:50.153828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2e996a-1721-4dee-b761-49fd74cc8ea4 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:50.171237 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 350/702] 10.222.0.62 () {60 vars in 1132 bytes} [Fri Sep 23 13:43:49 2022] POST /identity/v3/users => generated 254 bytes in 366 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:43:50.180369 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 353/703] 10.222.0.62 () {56 vars in 1133 bytes} [Fri Sep 23 13:43:49 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:50.186018 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0cd721-df7f-432c-960f-d2da6f264204 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:50.215363 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:50.308519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba81de91-846c-4c7c-8515-a9852b7aa793 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:50.309146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba81de91-846c-4c7c-8515-a9852b7aa793 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:50.309615 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba81de91-846c-4c7c-8515-a9852b7aa793 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:50.311293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 354/704] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:43:50.319794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c3ca7c-79de-41c2-a18b-82609afb0e5a None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:50.320205 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c3ca7c-79de-41c2-a18b-82609afb0e5a None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:50.320573 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c3ca7c-79de-41c2-a18b-82609afb0e5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:50.328185 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0cd721-df7f-432c-960f-d2da6f264204 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:50.353899 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-b3c3ca7c-79de-41c2-a18b-82609afb0e5a None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:50.355284 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-b3c3ca7c-79de-41c2-a18b-82609afb0e5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:50.370153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-926f5347-6a26-41ca-9dee-6fbc2528f955 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:50.370799 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926f5347-6a26-41ca-9dee-6fbc2528f955 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:50.371082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926f5347-6a26-41ca-9dee-6fbc2528f955 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:50.371357 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926f5347-6a26-41ca-9dee-6fbc2528f955 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:50.452955 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926f5347-6a26-41ca-9dee-6fbc2528f955 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:50.454574 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-926f5347-6a26-41ca-9dee-6fbc2528f955 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:50.466470 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:50.494322 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 355/705] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:50.570682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 351/706] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:50.575866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c3ca7c-79de-41c2-a18b-82609afb0e5a None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:50.702705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c3ca7c-79de-41c2-a18b-82609afb0e5a None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:50.726758 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:50.737055 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28928c94-16c2-46f2-a032-a556a1c4fee1 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:50.737341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28928c94-16c2-46f2-a032-a556a1c4fee1 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:50.737615 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28928c94-16c2-46f2-a032-a556a1c4fee1 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:50.740617 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 352/707] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:50.745667 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0876c7c3-3505-4cbd-94f9-5ef90e2f0733 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:50.745942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0876c7c3-3505-4cbd-94f9-5ef90e2f0733 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:50.746207 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0876c7c3-3505-4cbd-94f9-5ef90e2f0733 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:50.753358 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbdb1a1-8b44-43d7-8caa-1a2c15dcbaa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:50.756483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbdb1a1-8b44-43d7-8caa-1a2c15dcbaa0 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:50.757004 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbdb1a1-8b44-43d7-8caa-1a2c15dcbaa0 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:50.757482 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfbdb1a1-8b44-43d7-8caa-1a2c15dcbaa0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:50.767062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-0876c7c3-3505-4cbd-94f9-5ef90e2f0733 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:50.770111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-0876c7c3-3505-4cbd-94f9-5ef90e2f0733 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:50.770564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbdb1a1-8b44-43d7-8caa-1a2c15dcbaa0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:50.773907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfbdb1a1-8b44-43d7-8caa-1a2c15dcbaa0 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:50.791778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-cfbdb1a1-8b44-43d7-8caa-1a2c15dcbaa0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:50.791778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:50.791778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:50.791778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:50.791778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:50.791778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:50.791778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:50.791778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:50.791778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:50.791778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:50.791778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:50.791778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:50.791778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:50.791778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:50.791778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:50.791778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:50.792794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:50.792794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:50.792794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:50.792794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:50.792794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:50.792794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:50.792794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:50.792794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:50.792794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:50.792794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:50.792794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:50.792794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:50.792794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:50.792794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:50.792794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:50.792794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:50.792794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:50.793738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:50.793738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:50.793738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:50.795560 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 356/708] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:50.796539 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbdb1a1-8b44-43d7-8caa-1a2c15dcbaa0 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:50.848511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 353/709] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 23 13:43:50.850233 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0876c7c3-3505-4cbd-94f9-5ef90e2f0733 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:50.917851 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:50.963127 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfbdb1a1-8b44-43d7-8caa-1a2c15dcbaa0 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:51.029960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0876c7c3-3505-4cbd-94f9-5ef90e2f0733 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:51.053547 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01da8ff-f0d0-4333-852f-fd25bf33c667 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:51.054585 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01da8ff-f0d0-4333-852f-fd25bf33c667 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:51.055086 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01da8ff-f0d0-4333-852f-fd25bf33c667 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:51.055575 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01da8ff-f0d0-4333-852f-fd25bf33c667 None admin] PATH_INFO: `/v3/services/network` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:51.056369 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01da8ff-f0d0-4333-852f-fd25bf33c667 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:51.058425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01da8ff-f0d0-4333-852f-fd25bf33c667 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:51.080240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-b01da8ff-f0d0-4333-852f-fd25bf33c667 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 23 13:43:51.080240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:51.080240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:43:51.080240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:43:51.080240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:43:51.080240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:43:51.080240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:43:51.080240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:43:51.080240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 23 13:43:51.080240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:51.080240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:43:51.080240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:51.080240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:51.080240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:51.080240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:51.080240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:51.080240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:51.081351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:51.081351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:51.081351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:51.081351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:51.081351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:51.081351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:51.081351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:43:51.081351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:43:51.081351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:43:51.081351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:43:51.081351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:51.081351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:51.081351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:51.081351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:51.081351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:51.081351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:51.082279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:51.082279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:51.082279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:51.082279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:51.082279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:51.082279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:51.082279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:51.082279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:51.082279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:51.082279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:51.082279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:43:51.082279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:43:51.082279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 23 13:43:51.082279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:51.093122 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 357/710] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:50 2022] GET /identity/v3/services/network => generated 88 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:51.096444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0040eb9-b960-46cd-81e6-89a61a01ce10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:51.097827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01da8ff-f0d0-4333-852f-fd25bf33c667 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:51.100326 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0040eb9-b960-46cd-81e6-89a61a01ce10 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:51.103149 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0040eb9-b960-46cd-81e6-89a61a01ce10 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:51.104469 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0040eb9-b960-46cd-81e6-89a61a01ce10 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:51.105394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0040eb9-b960-46cd-81e6-89a61a01ce10 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:51.109936 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0040eb9-b960-46cd-81e6-89a61a01ce10 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:51.111583 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0040eb9-b960-46cd-81e6-89a61a01ce10 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:51.140576 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 354/711] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:51.141436 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0040eb9-b960-46cd-81e6-89a61a01ce10 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:51.164707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:51.259788 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01da8ff-f0d0-4333-852f-fd25bf33c667 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:51.262554 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:51.269714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0040eb9-b960-46cd-81e6-89a61a01ce10 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:51.304787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a40a05f-7ec3-448e-a889-44555efdabd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:51.305637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a40a05f-7ec3-448e-a889-44555efdabd7 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:51.306016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a40a05f-7ec3-448e-a889-44555efdabd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:51.306416 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a40a05f-7ec3-448e-a889-44555efdabd7 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:51.307112 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a40a05f-7ec3-448e-a889-44555efdabd7 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:51.307297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ba71b1-7ade-4af3-b30a-080b7aa5f667 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:51.308054 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ba71b1-7ade-4af3-b30a-080b7aa5f667 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:51.308486 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ba71b1-7ade-4af3-b30a-080b7aa5f667 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:51.308899 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ba71b1-7ade-4af3-b30a-080b7aa5f667 None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:51.309009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a40a05f-7ec3-448e-a889-44555efdabd7 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:51.309774 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ba71b1-7ade-4af3-b30a-080b7aa5f667 None admin] RBAC: Adding query filter params (name=network) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:51.310218 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ba71b1-7ade-4af3-b30a-080b7aa5f667 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:51.311913 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ba71b1-7ade-4af3-b30a-080b7aa5f667 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:51.317637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 358/712] 10.222.0.62 () {56 vars in 1112 bytes} [Fri Sep 23 13:43:51 2022] GET /identity/v3/services?name=network => generated 124 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:51.319316 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ba71b1-7ade-4af3-b30a-080b7aa5f667 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:51.321340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-1a40a05f-7ec3-448e-a889-44555efdabd7 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 23 13:43:51.321340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:51.321340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:43:51.321340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:43:51.321340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:43:51.321340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:43:51.321340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:43:51.321340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:43:51.321340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 23 13:43:51.321340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:51.321340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:43:51.321340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:51.321340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:51.321340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:51.321340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:51.321340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:51.321340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:51.322482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:51.322482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:51.322482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:51.322482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:51.322482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:51.322482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:51.322482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:43:51.322482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:43:51.322482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:43:51.322482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:43:51.322482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:51.322482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:51.322482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:51.322482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:51.322482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:51.322482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:51.323611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:51.323611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:51.323611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:51.323611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:51.323611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:51.323611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:51.323611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:51.323611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:51.323611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:51.323611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:51.323611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:43:51.323611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:43:51.323611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 23 13:43:51.323611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:51.325782 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 355/713] 10.222.0.62 () {56 vars in 1129 bytes} [Fri Sep 23 13:43:51 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:51.327154 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a40a05f-7ec3-448e-a889-44555efdabd7 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:51.435094 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a40a05f-7ec3-448e-a889-44555efdabd7 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:51.446009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ba71b1-7ade-4af3-b30a-080b7aa5f667 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:51.448839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:51.480253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77df8e83-15a9-44a3-9888-a409d23bbc50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:51.481144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77df8e83-15a9-44a3-9888-a409d23bbc50 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:51.481431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77df8e83-15a9-44a3-9888-a409d23bbc50 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:51.481701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77df8e83-15a9-44a3-9888-a409d23bbc50 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:51.482361 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77df8e83-15a9-44a3-9888-a409d23bbc50 None admin] RBAC: Adding query filter params (type=network) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:51.482961 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77df8e83-15a9-44a3-9888-a409d23bbc50 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:51.484656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77df8e83-15a9-44a3-9888-a409d23bbc50 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:51.488953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57843c78-19b5-41bc-b94b-4d18a6180090 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:51.490375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57843c78-19b5-41bc-b94b-4d18a6180090 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:51.490810 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57843c78-19b5-41bc-b94b-4d18a6180090 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:51.491194 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 356/714] 10.222.0.62 () {56 vars in 1112 bytes} [Fri Sep 23 13:43:51 2022] GET /identity/v3/services?type=network => generated 124 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:51.491532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57843c78-19b5-41bc-b94b-4d18a6180090 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:51.492139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77df8e83-15a9-44a3-9888-a409d23bbc50 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:51.500169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57843c78-19b5-41bc-b94b-4d18a6180090 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:51.502045 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57843c78-19b5-41bc-b94b-4d18a6180090 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:51.525100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-57843c78-19b5-41bc-b94b-4d18a6180090 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:51.525100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:51.525100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:51.525100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:51.525100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:51.525100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:51.525100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:51.525100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:51.525100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:51.525100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:51.525100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:51.525100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:51.525100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:43:51.525100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:43:51.525100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:43:51.526079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:43:51.526079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:51.526079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:51.526079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:51.526079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:51.526079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:51.526079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:51.526079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:51.526079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:51.526079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:51.526079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:51.526079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:51.526079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:51.526079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:51.526079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:51.526079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:51.526079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:51.526999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:43:51.526999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:43:51.526999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:43:51.526999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:43:51.526999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:43:51.526999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:43:51.526999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:51.526999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:51.531787 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 359/715] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:51.537092 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57843c78-19b5-41bc-b94b-4d18a6180090 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:51.545302 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:51.692903 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77df8e83-15a9-44a3-9888-a409d23bbc50 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:51.709766 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57843c78-19b5-41bc-b94b-4d18a6180090 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:51.744331 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0972491a-f78c-44f9-9ca6-997ae3152f72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:51.744986 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0972491a-f78c-44f9-9ca6-997ae3152f72 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:51.745268 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0972491a-f78c-44f9-9ca6-997ae3152f72 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:51.745543 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0972491a-f78c-44f9-9ca6-997ae3152f72 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:51.746171 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0972491a-f78c-44f9-9ca6-997ae3152f72 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:51.746517 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0972491a-f78c-44f9-9ca6-997ae3152f72 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:51.747164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c77b27-2cc0-4513-9ee9-4e2af110d625 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:51.747873 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0972491a-f78c-44f9-9ca6-997ae3152f72 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:51.748235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c77b27-2cc0-4513-9ee9-4e2af110d625 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:51.748707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c77b27-2cc0-4513-9ee9-4e2af110d625 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:51.749200 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c77b27-2cc0-4513-9ee9-4e2af110d625 None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:51.750086 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c77b27-2cc0-4513-9ee9-4e2af110d625 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:51.750769 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c77b27-2cc0-4513-9ee9-4e2af110d625 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:51.752408 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c77b27-2cc0-4513-9ee9-4e2af110d625 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:51.752769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 357/716] 10.222.0.62 () {56 vars in 1124 bytes} [Fri Sep 23 13:43:51 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:51.760188 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0972491a-f78c-44f9-9ca6-997ae3152f72 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:51.767589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 360/717] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:43:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:51.780869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c77b27-2cc0-4513-9ee9-4e2af110d625 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:51.824501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:51.895166 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:51.920241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0972491a-f78c-44f9-9ca6-997ae3152f72 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:51.957726 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c77b27-2cc0-4513-9ee9-4e2af110d625 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:51.995716 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67b68ca-0064-43b4-8b6c-5aad6e1e1936 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:51.996320 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67b68ca-0064-43b4-8b6c-5aad6e1e1936 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:51.996595 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67b68ca-0064-43b4-8b6c-5aad6e1e1936 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:51.996859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67b68ca-0064-43b4-8b6c-5aad6e1e1936 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:51.997579 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67b68ca-0064-43b4-8b6c-5aad6e1e1936 None admin] RBAC: Adding query filter params (role.id=727f0fa598e54b9a929d2cf6277c875e, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=f01ed94dbb384a178ad0a45a6f3d8c17) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:51.997880 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67b68ca-0064-43b4-8b6c-5aad6e1e1936 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:51.999324 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67b68ca-0064-43b4-8b6c-5aad6e1e1936 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:52.005661 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf21c00-684a-448c-90f3-c05652dc7a3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:52.006270 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf21c00-684a-448c-90f3-c05652dc7a3c None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:52.006573 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf21c00-684a-448c-90f3-c05652dc7a3c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:52.006844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf21c00-684a-448c-90f3-c05652dc7a3c None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:52.007470 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf21c00-684a-448c-90f3-c05652dc7a3c None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:52.007765 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf21c00-684a-448c-90f3-c05652dc7a3c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:52.009116 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf21c00-684a-448c-90f3-c05652dc7a3c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:52.018559 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 361/718] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:43:51 2022] GET /identity/v3/role_assignments?user.id=f01ed94dbb384a178ad0a45a6f3d8c17&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=727f0fa598e54b9a929d2cf6277c875e => generated 259 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:52.023513 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 358/719] 10.222.0.62 () {56 vars in 1124 bytes} [Fri Sep 23 13:43:51 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:52.530668 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee061f07-3b4f-426d-ad6f-e0956f376767 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:52.531153 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee061f07-3b4f-426d-ad6f-e0956f376767 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:52.531153 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee061f07-3b4f-426d-ad6f-e0956f376767 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:52.532738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 362/720] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:52.543922 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f4177f-7ff5-4c59-bc9a-c0908df4fa8f None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:52.544372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f4177f-7ff5-4c59-bc9a-c0908df4fa8f None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:52.544445 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f4177f-7ff5-4c59-bc9a-c0908df4fa8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:52.567889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-87f4177f-7ff5-4c59-bc9a-c0908df4fa8f None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:52.569099 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-87f4177f-7ff5-4c59-bc9a-c0908df4fa8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:52.657019 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 359/721] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:53.010591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee061f07-3b4f-426d-ad6f-e0956f376767 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:53.122404 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:53.145125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee061f07-3b4f-426d-ad6f-e0956f376767 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:53.224418 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00737ca-f33c-4f75-9041-17f1961ce5a7 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:53.224988 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00737ca-f33c-4f75-9041-17f1961ce5a7 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:53.225467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00737ca-f33c-4f75-9041-17f1961ce5a7 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:53.227361 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 360/722] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:43:53.233362 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf87f69-140f-4c0d-9cfd-5482d2ecac4f None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:53.233641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf87f69-140f-4c0d-9cfd-5482d2ecac4f None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:53.233912 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf87f69-140f-4c0d-9cfd-5482d2ecac4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:53.253735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9033825d-bb63-44f7-b0b5-fc155a5951ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:53.254831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9033825d-bb63-44f7-b0b5-fc155a5951ac None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:53.255442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9033825d-bb63-44f7-b0b5-fc155a5951ac None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:53.255918 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9033825d-bb63-44f7-b0b5-fc155a5951ac None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:53.268359 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9033825d-bb63-44f7-b0b5-fc155a5951ac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:53.272172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-ddf87f69-140f-4c0d-9cfd-5482d2ecac4f None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:53.273406 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ddf87f69-140f-4c0d-9cfd-5482d2ecac4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:53.274071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-9033825d-bb63-44f7-b0b5-fc155a5951ac None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:43:53.274071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:53.274071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:53.274071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:53.274071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:53.274071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:53.274071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:53.274071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:53.274071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:53.274071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:53.274071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:53.274071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:53.274071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:53.274071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:53.274071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:53.274071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:53.275012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:53.275012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:53.275012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:53.275012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:53.275012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:53.275012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:53.275012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:53.275012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:53.275012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:53.275012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:53.275012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:53.275012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:53.275012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:53.275012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:53.275012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:53.275012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:53.275857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:53.275857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:53.275857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:53.275857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:53.275857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:53.275857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:53.275857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:53.275857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:53.275857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:53.275857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:53.275857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:43:53.275857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:53.279455 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 363/723] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:43:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:53.284932 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9033825d-bb63-44f7-b0b5-fc155a5951ac None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:53.409309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 361/724] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:53.447910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9033825d-bb63-44f7-b0b5-fc155a5951ac None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:53.483582 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c12925-1a6c-4fd0-aca4-3daab190459f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:53.484557 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c12925-1a6c-4fd0-aca4-3daab190459f None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:53.485039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c12925-1a6c-4fd0-aca4-3daab190459f None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:53.485527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c12925-1a6c-4fd0-aca4-3daab190459f None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:53.486419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c12925-1a6c-4fd0-aca4-3daab190459f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:53.486939 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c12925-1a6c-4fd0-aca4-3daab190459f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:53.488486 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c12925-1a6c-4fd0-aca4-3daab190459f None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:53.498519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 364/725] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:53.506911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf87f69-140f-4c0d-9cfd-5482d2ecac4f None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:53.524927 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:53.634688 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf87f69-140f-4c0d-9cfd-5482d2ecac4f None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:53.677980 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556925b9-0da0-4322-bcb5-e750a3703193 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:53.678632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556925b9-0da0-4322-bcb5-e750a3703193 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:53.678906 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556925b9-0da0-4322-bcb5-e750a3703193 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:53.679172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556925b9-0da0-4322-bcb5-e750a3703193 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:53.695256 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556925b9-0da0-4322-bcb5-e750a3703193 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:53.696667 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556925b9-0da0-4322-bcb5-e750a3703193 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:53.717038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-556925b9-0da0-4322-bcb5-e750a3703193 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:53.717038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:53.717038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:53.717038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:53.717038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:53.717038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:53.717038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:53.717038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:53.717038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:53.717038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:53.717038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:53.717038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:53.717038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:53.717038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:53.717038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:53.717038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:53.717989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:53.717989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:53.717989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:53.717989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:53.717989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:53.717989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:53.717989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:53.717989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:53.717989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:53.717989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:53.717989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:53.717989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:53.717989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:53.717989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:53.717989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:53.717989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:53.717989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:53.719976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:53.719976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:53.719976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:53.719976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 362/726] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:53.723759 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c12925-1a6c-4fd0-aca4-3daab190459f None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:53.733527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:53.754380 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556925b9-0da0-4322-bcb5-e750a3703193 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:53.865425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c12925-1a6c-4fd0-aca4-3daab190459f None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:53.888907 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:53.928051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee4c376-12d4-4ea5-82e9-d61dc78d9347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:53.928671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee4c376-12d4-4ea5-82e9-d61dc78d9347 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:53.928938 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee4c376-12d4-4ea5-82e9-d61dc78d9347 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:53.929203 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee4c376-12d4-4ea5-82e9-d61dc78d9347 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:53.929849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee4c376-12d4-4ea5-82e9-d61dc78d9347 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:53.930146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee4c376-12d4-4ea5-82e9-d61dc78d9347 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:53.931536 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eee4c376-12d4-4ea5-82e9-d61dc78d9347 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:53.931977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556925b9-0da0-4322-bcb5-e750a3703193 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:53.960759 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 365/727] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:53.966072 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee4c376-12d4-4ea5-82e9-d61dc78d9347 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:53.970620 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6a3c5d-d448-4422-be5d-59b1c00450c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:53.971641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6a3c5d-d448-4422-be5d-59b1c00450c3 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:53.972122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6a3c5d-d448-4422-be5d-59b1c00450c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:53.972601 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6a3c5d-d448-4422-be5d-59b1c00450c3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:53.984425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6a3c5d-d448-4422-be5d-59b1c00450c3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:53.990417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-6e6a3c5d-d448-4422-be5d-59b1c00450c3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:53.990417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:53.990417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:53.990417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:53.990417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:53.990417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:53.990417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:53.990417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:53.990417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:53.990417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:53.990417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:53.990417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:53.990417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:53.990417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:53.990417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:53.990417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:53.991366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:53.991366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:53.991366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:53.991366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:53.991366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:53.991366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:53.991366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:53.991366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:53.991366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:53.991366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:53.991366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:53.991366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:53.991366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:53.991366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:53.991366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:53.991366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:53.992198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:53.992198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:53.992198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:53.992198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:53.992198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:53.992198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:53.992198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:53.992198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:53.992198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:53.992198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:53.992198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:43:53.992198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:53.995121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 363/728] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:43:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:43:54.000692 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6a3c5d-d448-4422-be5d-59b1c00450c3 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:54.128785 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6a3c5d-d448-4422-be5d-59b1c00450c3 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:54.148585 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:54.160653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:54.163137 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdf7aa5-dd93-4a25-ab27-e4533b624696 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:54.164005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdf7aa5-dd93-4a25-ab27-e4533b624696 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:54.165168 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdf7aa5-dd93-4a25-ab27-e4533b624696 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:54.166617 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdf7aa5-dd93-4a25-ab27-e4533b624696 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:54.167463 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdf7aa5-dd93-4a25-ab27-e4533b624696 None admin] RBAC: Adding query filter params (name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:54.169545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdf7aa5-dd93-4a25-ab27-e4533b624696 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:54.172699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdf7aa5-dd93-4a25-ab27-e4533b624696 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:54.190646 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 364/729] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:43:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:54.199350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdf7aa5-dd93-4a25-ab27-e4533b624696 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:54.219075 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eee4c376-12d4-4ea5-82e9-d61dc78d9347 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:54.260701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e33745-976b-4450-a057-b47f35e366d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:54.262029 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e33745-976b-4450-a057-b47f35e366d4 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:54.262704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e33745-976b-4450-a057-b47f35e366d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:54.262978 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e33745-976b-4450-a057-b47f35e366d4 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:54.274447 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e33745-976b-4450-a057-b47f35e366d4 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:54.276618 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e33745-976b-4450-a057-b47f35e366d4 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:54.291941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-c8e33745-976b-4450-a057-b47f35e366d4 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 23 13:43:54.291941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:54.291941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:54.291941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:54.291941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:54.291941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:54.291941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:54.291941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:54.291941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:54.291941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:54.291941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:54.291941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:54.291941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:43:54.291941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:43:54.291941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:43:54.291941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:43:54.292892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:54.292892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:54.292892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:43:54.292892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:54.292892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:43:54.292892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:54.292892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:54.292892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:54.292892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:54.292892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:54.292892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:54.292892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:54.292892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:54.292892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:54.292892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:54.292892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:54.292892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:54.292892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:54.293838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:54.293838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:54.293838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:43:54.293838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:43:54.293838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:43:54.293838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:43:54.293838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:43:54.293838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:43:54.293838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 23 13:43:54.293838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:54.298685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 366/730] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:43:53 2022] GET /identity/v3/users/swift => generated 83 bytes in 333 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:43:54.300101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f6b839-3d28-4e7a-b95c-0833ae18323c None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:54.300732 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f6b839-3d28-4e7a-b95c-0833ae18323c None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:54.301218 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f6b839-3d28-4e7a-b95c-0833ae18323c None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:54.304238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 367/731] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:54.305320 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f6b839-3d28-4e7a-b95c-0833ae18323c None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:54.360508 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdf7aa5-dd93-4a25-ab27-e4533b624696 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:54.410226 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d7c595-d44f-4941-9cc4-fed87c6b680c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:54.411727 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d7c595-d44f-4941-9cc4-fed87c6b680c None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:54.412298 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d7c595-d44f-4941-9cc4-fed87c6b680c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:54.412435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d7c595-d44f-4941-9cc4-fed87c6b680c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:54.424005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d7c595-d44f-4941-9cc4-fed87c6b680c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:54.425204 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d7c595-d44f-4941-9cc4-fed87c6b680c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:54.441954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-d3d7c595-d44f-4941-9cc4-fed87c6b680c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:54.441954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:54.441954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:54.441954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:54.441954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:54.441954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:54.441954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:54.441954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:54.441954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:54.441954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:54.441954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:54.441954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:54.441954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:54.441954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:54.441954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:54.441954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:54.443271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:54.443271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:54.443271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:54.443271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:54.443271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:54.443271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:54.443271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:54.443271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:54.443271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:54.443271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:54.443271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:54.443271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:54.443271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:54.443271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:54.443271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:54.443271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:54.443271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:54.444176 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:54.444176 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:54.444176 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:54.449066 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 365/732] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:54.450583 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e15f30-3836-4e6a-a31e-4c5ea2d5a216 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:54.450785 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e15f30-3836-4e6a-a31e-4c5ea2d5a216 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:54.450993 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e15f30-3836-4e6a-a31e-4c5ea2d5a216 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:54.460642 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f6b839-3d28-4e7a-b95c-0833ae18323c None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:54.470809 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-85e15f30-3836-4e6a-a31e-4c5ea2d5a216 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:54.471992 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-85e15f30-3836-4e6a-a31e-4c5ea2d5a216 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:54.496765 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:54.505228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:54.508702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdd00a5-21cf-4a63-809d-4106315a340e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:54.509314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdd00a5-21cf-4a63-809d-4106315a340e None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:54.512727 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdd00a5-21cf-4a63-809d-4106315a340e None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:54.512727 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdd00a5-21cf-4a63-809d-4106315a340e None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:54.512727 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdd00a5-21cf-4a63-809d-4106315a340e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:54.512727 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdd00a5-21cf-4a63-809d-4106315a340e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:54.512727 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdd00a5-21cf-4a63-809d-4106315a340e None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:54.530723 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 368/733] 10.222.0.62 () {56 vars in 1135 bytes} [Fri Sep 23 13:43:54 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:54.531562 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdd00a5-21cf-4a63-809d-4106315a340e None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:54.548545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 366/734] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:54.555803 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1344b14d-6ade-460e-81be-aec74323fce1 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:54.556095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1344b14d-6ade-460e-81be-aec74323fce1 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:54.561890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1344b14d-6ade-460e-81be-aec74323fce1 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:54.561890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 367/735] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:54 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:54.561890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1344b14d-6ade-460e-81be-aec74323fce1 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:54.653290 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdd00a5-21cf-4a63-809d-4106315a340e None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:54.691400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1344b14d-6ade-460e-81be-aec74323fce1 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:54.698547 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce8fb2b-72a8-4845-8166-e2e4415f8986 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:54.699164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce8fb2b-72a8-4845-8166-e2e4415f8986 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:54.699444 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce8fb2b-72a8-4845-8166-e2e4415f8986 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:54.699716 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce8fb2b-72a8-4845-8166-e2e4415f8986 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:54.700362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce8fb2b-72a8-4845-8166-e2e4415f8986 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:54.701163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce8fb2b-72a8-4845-8166-e2e4415f8986 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:54.702571 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce8fb2b-72a8-4845-8166-e2e4415f8986 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:54.719041 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 369/736] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:54.720620 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f879762a-82e5-4d6c-acaa-61580a9a4c88 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:54.720907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f879762a-82e5-4d6c-acaa-61580a9a4c88 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:54.721184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f879762a-82e5-4d6c-acaa-61580a9a4c88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:54.734447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1c2dac-51e7-4c2f-b30e-1e838e95fd1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:54.735609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1c2dac-51e7-4c2f-b30e-1e838e95fd1b None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:54.735895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1c2dac-51e7-4c2f-b30e-1e838e95fd1b None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:54.736170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1c2dac-51e7-4c2f-b30e-1e838e95fd1b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:54.742255 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-f879762a-82e5-4d6c-acaa-61580a9a4c88 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:54.744350 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f879762a-82e5-4d6c-acaa-61580a9a4c88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:54.782601 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:54.789170 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:54.822718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 370/737] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:54.823818 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f879762a-82e5-4d6c-acaa-61580a9a4c88 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:54.831234 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1c2dac-51e7-4c2f-b30e-1e838e95fd1b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:54.834279 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1c2dac-51e7-4c2f-b30e-1e838e95fd1b None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:54.935532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f879762a-82e5-4d6c-acaa-61580a9a4c88 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:54.963373 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 368/738] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:54.964234 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1c2dac-51e7-4c2f-b30e-1e838e95fd1b None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:54.979502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a48f834-b1af-4f68-87e6-bde8397fb63d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:54.980513 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a48f834-b1af-4f68-87e6-bde8397fb63d None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:54.980796 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a48f834-b1af-4f68-87e6-bde8397fb63d None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:54.981218 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a48f834-b1af-4f68-87e6-bde8397fb63d None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:54.988561 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a48f834-b1af-4f68-87e6-bde8397fb63d None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:54.990144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a48f834-b1af-4f68-87e6-bde8397fb63d None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:55.001860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-4a48f834-b1af-4f68-87e6-bde8397fb63d None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 23 13:43:55.001860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:55.001860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:55.001860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:55.001860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:55.001860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:55.001860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:55.001860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:55.001860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:55.001860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:55.001860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:55.001860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:55.001860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:43:55.001860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:43:55.001860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:43:55.001860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:43:55.002805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:55.002805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:55.002805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:43:55.002805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:55.002805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:43:55.002805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:55.002805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:55.002805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:55.002805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:55.002805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:55.002805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:55.002805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:55.002805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:55.002805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:55.002805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:55.002805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:55.002805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:55.002805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:55.003737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:55.003737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:55.003737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:43:55.003737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:43:55.003737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:43:55.003737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:43:55.003737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:43:55.003737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:43:55.003737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 23 13:43:55.003737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:55.003737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 371/739] 10.222.0.62 () {56 vars in 1117 bytes} [Fri Sep 23 13:43:54 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:55.004921 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bfe84e-7d34-42b4-82c3-27d74f60e216 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:55.005031 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bfe84e-7d34-42b4-82c3-27d74f60e216 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:55.005377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bfe84e-7d34-42b4-82c3-27d74f60e216 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:55.010729 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 372/740] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:55.010729 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bfe84e-7d34-42b4-82c3-27d74f60e216 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:55.042384 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:55.049744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:55.078709 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1c2dac-51e7-4c2f-b30e-1e838e95fd1b None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:55.113982 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dcc29d9-14b7-4515-8340-d7b689385992 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:55.114694 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcc29d9-14b7-4515-8340-d7b689385992 None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:55.114980 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcc29d9-14b7-4515-8340-d7b689385992 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:55.115247 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcc29d9-14b7-4515-8340-d7b689385992 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:55.120232 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dcc29d9-14b7-4515-8340-d7b689385992 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:55.120232 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dcc29d9-14b7-4515-8340-d7b689385992 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:55.120467 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bfe84e-7d34-42b4-82c3-27d74f60e216 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:55.154646 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57897810-8546-4c2f-8061-98d686e46c66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:55.155309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57897810-8546-4c2f-8061-98d686e46c66 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:55.155588 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57897810-8546-4c2f-8061-98d686e46c66 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:55.155855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57897810-8546-4c2f-8061-98d686e46c66 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:55.166313 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57897810-8546-4c2f-8061-98d686e46c66 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:55.167802 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57897810-8546-4c2f-8061-98d686e46c66 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:55.182476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-57897810-8546-4c2f-8061-98d686e46c66 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:55.182476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:55.182476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:55.182476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:55.182476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:55.182476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:55.182476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:55.182476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:55.182476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:55.182476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:55.182476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:55.182476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:55.182476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:55.182476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:55.182476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:55.182476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:55.183643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:55.183643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:55.183643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:55.183643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:55.183643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:55.183643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:55.183643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:55.183643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:55.183643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:55.183643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:55.183643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:55.183643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:55.183643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:55.183643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:55.183643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:55.183643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:55.183643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:55.184541 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:55.184541 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:55.184541 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:55.184541 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 373/741] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:55.184541 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57897810-8546-4c2f-8061-98d686e46c66 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:55.225917 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 369/742] 10.222.0.62 () {60 vars in 1141 bytes} [Fri Sep 23 13:43:54 2022] POST /identity/v3/services => generated 240 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:43:55.227228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba7c598-cd90-4bd5-86c5-99866cd0f161 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:55.227517 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba7c598-cd90-4bd5-86c5-99866cd0f161 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:55.227790 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba7c598-cd90-4bd5-86c5-99866cd0f161 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:55.248330 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-9ba7c598-cd90-4bd5-86c5-99866cd0f161 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:55.249573 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9ba7c598-cd90-4bd5-86c5-99866cd0f161 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:55.292993 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:55.311276 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57897810-8546-4c2f-8061-98d686e46c66 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:55.351030 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6592bba-ba3c-4415-a759-b8976b40e263 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:55.351653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6592bba-ba3c-4415-a759-b8976b40e263 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:55.351928 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6592bba-ba3c-4415-a759-b8976b40e263 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:55.352198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6592bba-ba3c-4415-a759-b8976b40e263 None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:55.352948 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6592bba-ba3c-4415-a759-b8976b40e263 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:55.353253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6592bba-ba3c-4415-a759-b8976b40e263 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:55.354806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6592bba-ba3c-4415-a759-b8976b40e263 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:55.355075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 370/743] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:55 2022] POST /identity/v3/auth/tokens => generated 898 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 23 13:43:55.356264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba7c598-cd90-4bd5-86c5-99866cd0f161 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:55.379411 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 374/744] 10.222.0.62 () {56 vars in 1149 bytes} [Fri Sep 23 13:43:55 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:55.381139 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6592bba-ba3c-4415-a759-b8976b40e263 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:55.425515 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:55.476589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba7c598-cd90-4bd5-86c5-99866cd0f161 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:55.512529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6592bba-ba3c-4415-a759-b8976b40e263 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:55.513517 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2354cb27-9001-41df-9771-dec633cc0abb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:55.514219 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2354cb27-9001-41df-9771-dec633cc0abb None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:55.514619 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2354cb27-9001-41df-9771-dec633cc0abb None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:55.514955 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2354cb27-9001-41df-9771-dec633cc0abb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:55.525990 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2354cb27-9001-41df-9771-dec633cc0abb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:43:55.532267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-2354cb27-9001-41df-9771-dec633cc0abb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:43:55.532267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:55.532267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:55.532267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:55.532267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:55.532267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:55.532267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:55.532267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:55.532267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:55.532267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:55.532267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:55.532267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:55.532267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:43:55.532267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:43:55.532267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:43:55.532267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:43:55.533198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:43:55.533198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:43:55.533198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:43:55.533198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:43:55.533198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:55.533198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:55.533198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:55.533198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:55.533198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:55.533198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:55.533198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:55.533198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:55.533198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:55.533198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:55.533198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:55.533198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:55.534236 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:55.534236 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:55.534236 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:55.534236 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:55.534236 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:43:55.534236 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:43:55.534236 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:43:55.534236 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:43:55.534236 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:43:55.534236 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:43:55.534236 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:43:55.534236 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:55.534236 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 371/745] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:43:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:55.539020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2354cb27-9001-41df-9771-dec633cc0abb None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:55.551973 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8f761d-e043-462d-9b4a-e34f394b69a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:55.552591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8f761d-e043-462d-9b4a-e34f394b69a7 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:55.552866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8f761d-e043-462d-9b4a-e34f394b69a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:55.553142 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8f761d-e043-462d-9b4a-e34f394b69a7 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:55.553793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8f761d-e043-462d-9b4a-e34f394b69a7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:55.554514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8f761d-e043-462d-9b4a-e34f394b69a7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:55.556241 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8f761d-e043-462d-9b4a-e34f394b69a7 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:55.572601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 375/746] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:55.573734 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8f761d-e043-462d-9b4a-e34f394b69a7 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:55.578513 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:55.653106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2354cb27-9001-41df-9771-dec633cc0abb None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:55.668657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:55.691683 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8acd57-77b5-47cc-a224-a55bdd23970b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:55.692812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8acd57-77b5-47cc-a224-a55bdd23970b None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:55.692812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8acd57-77b5-47cc-a224-a55bdd23970b None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:55.693035 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8acd57-77b5-47cc-a224-a55bdd23970b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:55.693514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8f761d-e043-462d-9b4a-e34f394b69a7 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:55.730424 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d55b63b-4ede-486b-ad53-8e5e1e653c54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:55.730976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d55b63b-4ede-486b-ad53-8e5e1e653c54 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:55.731258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d55b63b-4ede-486b-ad53-8e5e1e653c54 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:55.731531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d55b63b-4ede-486b-ad53-8e5e1e653c54 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:55.732193 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d55b63b-4ede-486b-ad53-8e5e1e653c54 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:55.732494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d55b63b-4ede-486b-ad53-8e5e1e653c54 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:55.733891 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d55b63b-4ede-486b-ad53-8e5e1e653c54 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:55.743431 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 376/747] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:43:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:55.744220 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d55b63b-4ede-486b-ad53-8e5e1e653c54 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:55.775225 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8acd57-77b5-47cc-a224-a55bdd23970b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:55.776691 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8acd57-77b5-47cc-a224-a55bdd23970b None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:55.815753 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:55.851445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d55b63b-4ede-486b-ad53-8e5e1e653c54 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:55.888521 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb112656-268b-4c88-922e-110970fa6f5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:55.889029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 372/748] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:43:55 2022] GET /identity/v3/auth/tokens => generated 898 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 23 13:43:55.889334 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb112656-268b-4c88-922e-110970fa6f5d None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:55.889767 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb112656-268b-4c88-922e-110970fa6f5d None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:55.890135 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb112656-268b-4c88-922e-110970fa6f5d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:55.890477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8acd57-77b5-47cc-a224-a55bdd23970b None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:55.899743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb112656-268b-4c88-922e-110970fa6f5d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:55.901192 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb112656-268b-4c88-922e-110970fa6f5d None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:55.907210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:55.913147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-eb112656-268b-4c88-922e-110970fa6f5d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:55.913147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:55.913147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:55.913147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:55.913147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:55.913147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:55.913147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:55.913147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:55.913147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:55.913147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:55.913147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:55.913147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:55.913147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:43:55.913147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:43:55.913147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:43:55.918302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:43:55.918302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:55.918302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:55.918302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:55.918302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:55.918302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:55.918302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:55.918302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:55.918302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:55.918302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:55.918302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:55.918302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:55.918302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:55.918302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:55.918302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:55.918302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:55.918302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:55.919240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:43:55.919240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:43:55.919240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:43:55.919240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:43:55.919240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:43:55.919240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:43:55.919240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:55.919240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:55.919240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 377/749] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:55.919240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb112656-268b-4c88-922e-110970fa6f5d None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:56.004343 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8acd57-77b5-47cc-a224-a55bdd23970b None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:56.022017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb112656-268b-4c88-922e-110970fa6f5d None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:56.042772 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cc7abe-c023-417a-ad24-2ae3201d75a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:56.043871 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cc7abe-c023-417a-ad24-2ae3201d75a1 None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:56.044363 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cc7abe-c023-417a-ad24-2ae3201d75a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:56.044705 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cc7abe-c023-417a-ad24-2ae3201d75a1 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:56.045428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cc7abe-c023-417a-ad24-2ae3201d75a1 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:56.046829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cc7abe-c023-417a-ad24-2ae3201d75a1 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:56.048750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:56.058512 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db94fb4f-48ac-4b08-b8c8-68264d61c840 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:56.059132 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db94fb4f-48ac-4b08-b8c8-68264d61c840 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:56.059406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db94fb4f-48ac-4b08-b8c8-68264d61c840 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:56.059672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db94fb4f-48ac-4b08-b8c8-68264d61c840 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:56.069345 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db94fb4f-48ac-4b08-b8c8-68264d61c840 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:56.070795 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db94fb4f-48ac-4b08-b8c8-68264d61c840 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:56.084651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-db94fb4f-48ac-4b08-b8c8-68264d61c840 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:56.084651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:56.084651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:56.084651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:56.084651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:56.084651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:56.084651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:56.084651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:56.084651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:56.084651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:56.084651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:56.084651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:56.084651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:56.084651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:56.084651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:56.084651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:56.085762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:56.085762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:56.085762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:56.085762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:56.085762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:56.085762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:56.085762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:56.085762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:56.085762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:56.085762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:56.085762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:56.085762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:56.085762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:56.085762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:56.085762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:56.085762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:56.085762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:56.086839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:56.086839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:56.086839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:56.086839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 378/750] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:56.086839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db94fb4f-48ac-4b08-b8c8-68264d61c840 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:56.186496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db94fb4f-48ac-4b08-b8c8-68264d61c840 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:56.190058 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 373/751] 10.222.0.62 () {60 vars in 1141 bytes} [Fri Sep 23 13:43:55 2022] POST /identity/v3/services => generated 251 bytes in 300 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:43:56.190966 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cc7abe-c023-417a-ad24-2ae3201d75a1 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:56.269183 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e30909-50a1-45ab-b783-ed3ddc966347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:56.269769 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e30909-50a1-45ab-b783-ed3ddc966347 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:56.270073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e30909-50a1-45ab-b783-ed3ddc966347 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:56.270889 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e30909-50a1-45ab-b783-ed3ddc966347 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:56.288075 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e30909-50a1-45ab-b783-ed3ddc966347 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:56.289856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e30909-50a1-45ab-b783-ed3ddc966347 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:56.301734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:56.313532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-a1e30909-50a1-45ab-b783-ed3ddc966347 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:56.313532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:56.313532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:56.313532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:56.313532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:56.313532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:56.313532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:56.313532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:56.313532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:56.313532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:56.313532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:56.313532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:56.313532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:56.313532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:56.313532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:56.313532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:56.323067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:56.323067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:56.323067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:56.323067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:56.323067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:56.323067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:56.323067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:56.323067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:56.323067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:56.323067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:56.323067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:56.323067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:56.323067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:56.323067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:56.323067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:56.323067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:56.323067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:56.324001 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:56.324001 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:56.324001 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:56.324001 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 379/752] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:56.324001 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e30909-50a1-45ab-b783-ed3ddc966347 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:56.340079 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cc7abe-c023-417a-ad24-2ae3201d75a1 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:56.361377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:56.387470 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1837bf1a-9bd6-483e-8a53-1a8e0202f8bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:56.388831 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1837bf1a-9bd6-483e-8a53-1a8e0202f8bf None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:56.388831 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1837bf1a-9bd6-483e-8a53-1a8e0202f8bf None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:56.388831 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1837bf1a-9bd6-483e-8a53-1a8e0202f8bf None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:56.389299 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1837bf1a-9bd6-483e-8a53-1a8e0202f8bf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:56.389875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1837bf1a-9bd6-483e-8a53-1a8e0202f8bf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:56.391301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1837bf1a-9bd6-483e-8a53-1a8e0202f8bf None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:56.414356 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 374/753] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:43:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:56.414356 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1837bf1a-9bd6-483e-8a53-1a8e0202f8bf None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:56.458815 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e30909-50a1-45ab-b783-ed3ddc966347 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:56.499590 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96cd1bef-492d-49dd-b23c-64616eaf8c4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:56.500223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cd1bef-492d-49dd-b23c-64616eaf8c4b None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:56.500504 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cd1bef-492d-49dd-b23c-64616eaf8c4b None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:56.500773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cd1bef-492d-49dd-b23c-64616eaf8c4b None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:56.501440 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96cd1bef-492d-49dd-b23c-64616eaf8c4b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:56.501741 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96cd1bef-492d-49dd-b23c-64616eaf8c4b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:56.503201 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96cd1bef-492d-49dd-b23c-64616eaf8c4b None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:56.517092 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 380/754] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:56.518026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96cd1bef-492d-49dd-b23c-64616eaf8c4b None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:56.618913 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:56.635451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96cd1bef-492d-49dd-b23c-64616eaf8c4b None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:56.662309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1837bf1a-9bd6-483e-8a53-1a8e0202f8bf None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:56.671982 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d79c114-d7fa-4fc6-8bbc-9330a007cdf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:56.672779 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d79c114-d7fa-4fc6-8bbc-9330a007cdf7 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:56.673203 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d79c114-d7fa-4fc6-8bbc-9330a007cdf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:56.673639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d79c114-d7fa-4fc6-8bbc-9330a007cdf7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:56.674538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d79c114-d7fa-4fc6-8bbc-9330a007cdf7 None admin] RBAC: Adding query filter params (role.id=727f0fa598e54b9a929d2cf6277c875e, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=4b9269509f1d454eab7a3008c2ba0039) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:56.674986 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d79c114-d7fa-4fc6-8bbc-9330a007cdf7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:56.677091 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d79c114-d7fa-4fc6-8bbc-9330a007cdf7 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:56.683324 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 381/755] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:43:56 2022] GET /identity/v3/role_assignments?user.id=4b9269509f1d454eab7a3008c2ba0039&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=727f0fa598e54b9a929d2cf6277c875e => generated 259 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:56.684361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d79c114-d7fa-4fc6-8bbc-9330a007cdf7 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:56.702216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6673af26-9633-4afc-ac51-45210d52126e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:56.702812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6673af26-9633-4afc-ac51-45210d52126e None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:56.702893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6673af26-9633-4afc-ac51-45210d52126e None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:56.703155 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6673af26-9633-4afc-ac51-45210d52126e None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:56.703795 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6673af26-9633-4afc-ac51-45210d52126e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:56.704081 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6673af26-9633-4afc-ac51-45210d52126e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:56.705444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6673af26-9633-4afc-ac51-45210d52126e None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:56.708012 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:56.721788 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 375/756] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:56.734650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6673af26-9633-4afc-ac51-45210d52126e None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:56.853592 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d79c114-d7fa-4fc6-8bbc-9330a007cdf7 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:56.916796 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f632790-51a7-4213-a6a7-56d47a192f26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:56.917766 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f632790-51a7-4213-a6a7-56d47a192f26 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:56.918705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f632790-51a7-4213-a6a7-56d47a192f26 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:56.919282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f632790-51a7-4213-a6a7-56d47a192f26 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:56.926865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6673af26-9633-4afc-ac51-45210d52126e None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:56.937763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f632790-51a7-4213-a6a7-56d47a192f26 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:56.939605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f632790-51a7-4213-a6a7-56d47a192f26 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:56.960749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-5f632790-51a7-4213-a6a7-56d47a192f26 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 23 13:43:56.960749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:56.960749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:56.960749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:56.960749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:56.960749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:56.960749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:56.960749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:56.960749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:56.960749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:56.960749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:56.960749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:56.960749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:43:56.960749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:43:56.960749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:43:56.960749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:43:56.961743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:56.961743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:56.961743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:43:56.961743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:56.961743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:43:56.961743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:43:56.961743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:56.961743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:56.961743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:56.961743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:56.961743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:56.961743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:56.961743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:56.961743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:56.961743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:56.961743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:56.961743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:56.961743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:56.962743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:56.962743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:56.962743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:43:56.962743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:43:56.962743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:43:56.962743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:43:56.962743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:43:56.962743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:43:56.962743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 23 13:43:56.962743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:56.962743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 382/757] 10.222.0.62 () {56 vars in 1093 bytes} [Fri Sep 23 13:43:56 2022] GET /identity/v3/users/nova => generated 82 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:56.967251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f632790-51a7-4213-a6a7-56d47a192f26 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:56.969887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d86b768-6a1a-44eb-93f7-bd6c1faa25dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:56.970551 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d86b768-6a1a-44eb-93f7-bd6c1faa25dc None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:56.970832 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d86b768-6a1a-44eb-93f7-bd6c1faa25dc None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:56.971086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d86b768-6a1a-44eb-93f7-bd6c1faa25dc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:56.987681 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d86b768-6a1a-44eb-93f7-bd6c1faa25dc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:56.989716 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d86b768-6a1a-44eb-93f7-bd6c1faa25dc None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:56.994073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:57.009119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-9d86b768-6a1a-44eb-93f7-bd6c1faa25dc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:57.009119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:57.009119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:57.009119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:57.009119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:57.009119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:57.009119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:57.009119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:57.009119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:57.009119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:57.009119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:57.009119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:57.009119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:43:57.009119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:43:57.009119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:43:57.012444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:43:57.012444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:57.012444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:57.012444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:57.012444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:57.012444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:57.012444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:57.012444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:57.012444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:57.012444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:57.012444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:57.012444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:57.012444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:57.012444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:57.012444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:57.012444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:57.012444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:57.013314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:43:57.013314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:43:57.013314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:43:57.013314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:43:57.013314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:43:57.013314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:43:57.013314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:57.013314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:57.013314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 376/758] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:57.018157 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d86b768-6a1a-44eb-93f7-bd6c1faa25dc None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:57.078817 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:57.145213 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d86b768-6a1a-44eb-93f7-bd6c1faa25dc None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:57.219240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f632790-51a7-4213-a6a7-56d47a192f26 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:57.222114 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482dcd3b-1fa9-4431-bf5b-65daa752b34e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:57.222752 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482dcd3b-1fa9-4431-bf5b-65daa752b34e None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:57.223225 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482dcd3b-1fa9-4431-bf5b-65daa752b34e None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:57.223494 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482dcd3b-1fa9-4431-bf5b-65daa752b34e None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:57.224198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482dcd3b-1fa9-4431-bf5b-65daa752b34e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:57.224499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482dcd3b-1fa9-4431-bf5b-65daa752b34e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:57.225917 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482dcd3b-1fa9-4431-bf5b-65daa752b34e None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:57.256455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 377/759] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:43:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:57.263844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482dcd3b-1fa9-4431-bf5b-65daa752b34e None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:57.284515 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705b17a7-e2e9-481a-9614-7488ea6c2a77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:57.285124 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705b17a7-e2e9-481a-9614-7488ea6c2a77 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:57.285396 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705b17a7-e2e9-481a-9614-7488ea6c2a77 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:57.285870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705b17a7-e2e9-481a-9614-7488ea6c2a77 None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:57.286638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705b17a7-e2e9-481a-9614-7488ea6c2a77 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:57.286935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705b17a7-e2e9-481a-9614-7488ea6c2a77 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:57.288350 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705b17a7-e2e9-481a-9614-7488ea6c2a77 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:57.321376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 383/760] 10.222.0.62 () {56 vars in 1133 bytes} [Fri Sep 23 13:43:56 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 354 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:57.325771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705b17a7-e2e9-481a-9614-7488ea6c2a77 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:57.442700 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:57.444477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482dcd3b-1fa9-4431-bf5b-65daa752b34e None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:57.471025 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:57.491366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705b17a7-e2e9-481a-9614-7488ea6c2a77 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:57.515945 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c2fdf7-f9e6-4dc8-8b3d-13a9eccaced0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:57.516557 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c2fdf7-f9e6-4dc8-8b3d-13a9eccaced0 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:57.516838 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c2fdf7-f9e6-4dc8-8b3d-13a9eccaced0 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:57.517108 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c2fdf7-f9e6-4dc8-8b3d-13a9eccaced0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:57.518008 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c2fdf7-f9e6-4dc8-8b3d-13a9eccaced0 None admin] RBAC: Adding query filter params (role.id=809612fc6d62442cbcc179b5c4123465, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=b471d768cd814ff8b4f6bd35d21e24a8) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:57.518673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c2fdf7-f9e6-4dc8-8b3d-13a9eccaced0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:57.520655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c2fdf7-f9e6-4dc8-8b3d-13a9eccaced0 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:57.532967 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 378/761] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:43:57 2022] GET /identity/v3/role_assignments?user.id=b471d768cd814ff8b4f6bd35d21e24a8&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=809612fc6d62442cbcc179b5c4123465 => generated 259 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:57.549415 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2d9b73-dd23-4f60-82b3-d420554e25cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:57.550088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2d9b73-dd23-4f60-82b3-d420554e25cb None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:57.550391 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2d9b73-dd23-4f60-82b3-d420554e25cb None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:57.550736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2d9b73-dd23-4f60-82b3-d420554e25cb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:57.569341 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2d9b73-dd23-4f60-82b3-d420554e25cb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:57.570962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2d9b73-dd23-4f60-82b3-d420554e25cb None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:57.590251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb59503-c0a8-418f-9ce2-1d09349da3ca None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:57.590914 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb59503-c0a8-418f-9ce2-1d09349da3ca None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:57.591405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb59503-c0a8-418f-9ce2-1d09349da3ca None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:57.594803 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 379/762] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:43:57.597446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-8d2d9b73-dd23-4f60-82b3-d420554e25cb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:57.597446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:57.597446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:57.597446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:57.597446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:57.597446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:57.597446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:57.597446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:57.597446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:57.597446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:57.597446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:57.597446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:57.597446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:43:57.597446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:43:57.597446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:43:57.597446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:43:57.599400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:57.599400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:57.599400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:57.599400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:57.599400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:57.599400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:57.599400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:57.599400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:57.599400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:57.599400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:57.599400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:57.599400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:57.599400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:57.599400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:57.599400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:57.599400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:57.599400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:43:57.600282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:43:57.600282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:43:57.600282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:57.600282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 384/763] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:43:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 272 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:57.603198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f33e313-df24-439e-af8a-e732000a70dc None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:57.603662 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2d9b73-dd23-4f60-82b3-d420554e25cb None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:57.604097 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f33e313-df24-439e-af8a-e732000a70dc None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:57.604570 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f33e313-df24-439e-af8a-e732000a70dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:57.625017 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-6f33e313-df24-439e-af8a-e732000a70dc None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:57.626871 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-6f33e313-df24-439e-af8a-e732000a70dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:57.718520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2d9b73-dd23-4f60-82b3-d420554e25cb None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:57.742042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:57.761870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 380/764] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:57 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:43:57.779066 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ad6082-2e10-400e-9e66-ca627e7f0d2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:57.779686 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ad6082-2e10-400e-9e66-ca627e7f0d2e None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:57.779958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ad6082-2e10-400e-9e66-ca627e7f0d2e None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:57.780224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ad6082-2e10-400e-9e66-ca627e7f0d2e None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:57.780939 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ad6082-2e10-400e-9e66-ca627e7f0d2e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:57.781243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ad6082-2e10-400e-9e66-ca627e7f0d2e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:57.782967 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ad6082-2e10-400e-9e66-ca627e7f0d2e None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:57.801542 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 385/765] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:43:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:57.806674 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f33e313-df24-439e-af8a-e732000a70dc None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:57.958316 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f33e313-df24-439e-af8a-e732000a70dc None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:57.970405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:58.007986 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bced40e-77f3-40e5-87a2-c629f03ce8f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:58.008539 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bced40e-77f3-40e5-87a2-c629f03ce8f0 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:58.008813 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bced40e-77f3-40e5-87a2-c629f03ce8f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:58.009079 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bced40e-77f3-40e5-87a2-c629f03ce8f0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:58.019293 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bced40e-77f3-40e5-87a2-c629f03ce8f0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:58.020753 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bced40e-77f3-40e5-87a2-c629f03ce8f0 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:58.035558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-4bced40e-77f3-40e5-87a2-c629f03ce8f0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:58.035558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:58.035558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:58.035558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:58.035558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:58.035558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:58.035558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:58.035558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:58.035558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:58.035558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:58.035558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:58.035558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:58.035558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:43:58.035558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:43:58.035558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:43:58.036427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:43:58.036427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:58.036427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:58.036427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:58.036427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:58.036427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:58.036427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:58.036427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:58.036427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:58.036427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:58.036427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:58.036427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:58.036427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:58.036427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:58.036427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:58.036427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:58.036427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:58.037787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:43:58.037787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:43:58.037787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:43:58.037787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:43:58.037787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:43:58.037787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:43:58.037787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:43:58.037787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:58.037787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 381/766] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:58.042659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ad6082-2e10-400e-9e66-ca627e7f0d2e None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:58.047736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bced40e-77f3-40e5-87a2-c629f03ce8f0 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:58.182846 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bced40e-77f3-40e5-87a2-c629f03ce8f0 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:58.225913 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5530933-3266-4bb6-8de8-21dfca366917 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:58.226967 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5530933-3266-4bb6-8de8-21dfca366917 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:58.227607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5530933-3266-4bb6-8de8-21dfca366917 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:58.228897 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5530933-3266-4bb6-8de8-21dfca366917 None admin] PATH_INFO: `/v3/services/network` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:58.229689 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5530933-3266-4bb6-8de8-21dfca366917 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:58.232627 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5530933-3266-4bb6-8de8-21dfca366917 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:58.246395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-a5530933-3266-4bb6-8de8-21dfca366917 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 23 13:43:58.246395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:58.246395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:43:58.246395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:43:58.246395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:43:58.246395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:43:58.246395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:43:58.246395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:43:58.246395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 23 13:43:58.246395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:58.246395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:43:58.246395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:58.246395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:58.246395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:58.246395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:58.246395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:58.246395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:58.247411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:58.247411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:58.247411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:58.247411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:58.247411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:58.247411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:58.247411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:43:58.247411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:43:58.247411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:43:58.247411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:43:58.247411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:58.247411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:58.247411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:58.247411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:58.247411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:58.247411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:58.248244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:58.248244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:58.248244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:58.248244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:58.248244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:58.248244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:58.248244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:58.248244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:58.248244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:58.248244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:58.248244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:43:58.248244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:43:58.248244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 23 13:43:58.248244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:43:58.255472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 382/767] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:43:58 2022] GET /identity/v3/services/network => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:43:58.258070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5530933-3266-4bb6-8de8-21dfca366917 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:58.295704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ad6082-2e10-400e-9e66-ca627e7f0d2e None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:58.343486 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:58.373060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d300df0-f32b-43e0-affe-36f6a3715387 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:58.374037 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d300df0-f32b-43e0-affe-36f6a3715387 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:58.374537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d300df0-f32b-43e0-affe-36f6a3715387 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:58.375004 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d300df0-f32b-43e0-affe-36f6a3715387 None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:58.375906 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d300df0-f32b-43e0-affe-36f6a3715387 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:58.377481 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d300df0-f32b-43e0-affe-36f6a3715387 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:58.379962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d300df0-f32b-43e0-affe-36f6a3715387 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:58.409779 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 386/768] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:43:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 368 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:58.416259 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d300df0-f32b-43e0-affe-36f6a3715387 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:58.460111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5530933-3266-4bb6-8de8-21dfca366917 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:58.498873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:58.511170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0ee89f-1062-4fd1-a558-0ae47be4866d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:58.511879 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0ee89f-1062-4fd1-a558-0ae47be4866d None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:58.512268 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0ee89f-1062-4fd1-a558-0ae47be4866d None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:58.512568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0ee89f-1062-4fd1-a558-0ae47be4866d None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:58.513179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0ee89f-1062-4fd1-a558-0ae47be4866d None admin] RBAC: Adding query filter params (name=network) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:58.513478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0ee89f-1062-4fd1-a558-0ae47be4866d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:58.515074 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0ee89f-1062-4fd1-a558-0ae47be4866d None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:58.519995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 383/769] 10.222.0.62 () {56 vars in 1112 bytes} [Fri Sep 23 13:43:58 2022] GET /identity/v3/services?name=network => generated 124 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:58.525958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0ee89f-1062-4fd1-a558-0ae47be4866d None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:58.589487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d300df0-f32b-43e0-affe-36f6a3715387 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:58.656188 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0ee89f-1062-4fd1-a558-0ae47be4866d None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:58.667803 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:58.691816 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1b7619-1fc4-4e4d-82ec-d52da7f1b7ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:58.693452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a57078d-fe68-46f6-b5d8-561f815cb1ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:58.694489 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1b7619-1fc4-4e4d-82ec-d52da7f1b7ea None admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:58.695392 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a57078d-fe68-46f6-b5d8-561f815cb1ff None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:58.695898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1b7619-1fc4-4e4d-82ec-d52da7f1b7ea None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:58.696379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a57078d-fe68-46f6-b5d8-561f815cb1ff None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:58.696893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1b7619-1fc4-4e4d-82ec-d52da7f1b7ea None admin] PATH_INFO: `/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/f01ed94dbb384a178ad0a45a6f3d8c17/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:58.697364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a57078d-fe68-46f6-b5d8-561f815cb1ff None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:58.699902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1b7619-1fc4-4e4d-82ec-d52da7f1b7ea None admin] RBAC: Authorizing `identity:create_grant(project_id=2a19fd4315ad4a6eb349dfe8cfa66c23, user_id=f01ed94dbb384a178ad0a45a6f3d8c17, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:58.700967 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a57078d-fe68-46f6-b5d8-561f815cb1ff None admin] RBAC: Adding query filter params (type=network) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:58.701480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a57078d-fe68-46f6-b5d8-561f815cb1ff None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:58.703143 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a57078d-fe68-46f6-b5d8-561f815cb1ff None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:58.704902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1b7619-1fc4-4e4d-82ec-d52da7f1b7ea None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:58.714699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 384/770] 10.222.0.62 () {56 vars in 1112 bytes} [Fri Sep 23 13:43:58 2022] GET /identity/v3/services?type=network => generated 350 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:58.720575 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a57078d-fe68-46f6-b5d8-561f815cb1ff None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:58.789066 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 387/771] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:43:58 2022] PUT /identity/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/f01ed94dbb384a178ad0a45a6f3d8c17/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 373 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:43:58.815958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7361012-fa03-4133-a032-080c8d8a2c09 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:58.816284 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7361012-fa03-4133-a032-080c8d8a2c09 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:58.816505 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7361012-fa03-4133-a032-080c8d8a2c09 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:58.818464 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 388/772] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:58.824158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb5f0d8-0e98-459d-9c7e-93db9841b8ef None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:58.824437 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb5f0d8-0e98-459d-9c7e-93db9841b8ef None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:58.824692 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb5f0d8-0e98-459d-9c7e-93db9841b8ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:58.833016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a57078d-fe68-46f6-b5d8-561f815cb1ff None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:58.865958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-5cb5f0d8-0e98-459d-9c7e-93db9841b8ef None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:58.867360 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-5cb5f0d8-0e98-459d-9c7e-93db9841b8ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:58.873903 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a31b947-884f-4e22-8092-fffa4006fc4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:58.874942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a31b947-884f-4e22-8092-fffa4006fc4d None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:58.876454 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a31b947-884f-4e22-8092-fffa4006fc4d None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:58.876950 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a31b947-884f-4e22-8092-fffa4006fc4d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:58.877931 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a31b947-884f-4e22-8092-fffa4006fc4d None admin] RBAC: Adding query filter params (interface=public, service_id=e6800875e14e40e89af06383218f9211, region_id=RegionOne) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:58.878514 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a31b947-884f-4e22-8092-fffa4006fc4d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:58.880489 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a31b947-884f-4e22-8092-fffa4006fc4d None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:58.890734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 385/773] 10.222.0.62 () {56 vars in 1251 bytes} [Fri Sep 23 13:43:58 2022] GET /identity/v3/endpoints?service_id=e6800875e14e40e89af06383218f9211&interface=public®ion_id=RegionOne => generated 194 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:43:58.899992 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a31b947-884f-4e22-8092-fffa4006fc4d None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:58.931853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:59.020129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 389/774] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:58 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 23 13:43:59.029841 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a31b947-884f-4e22-8092-fffa4006fc4d None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:59.073318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65989c7e-5fcc-4cbf-8ff8-7e0a5f63b744 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:59.073848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65989c7e-5fcc-4cbf-8ff8-7e0a5f63b744 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:59.074170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65989c7e-5fcc-4cbf-8ff8-7e0a5f63b744 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:59.074464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65989c7e-5fcc-4cbf-8ff8-7e0a5f63b744 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:59.075072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65989c7e-5fcc-4cbf-8ff8-7e0a5f63b744 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:59.076424 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65989c7e-5fcc-4cbf-8ff8-7e0a5f63b744 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:59.087413 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 386/775] 10.222.0.62 () {56 vars in 1087 bytes} [Fri Sep 23 13:43:58 2022] GET /identity/v3/services => generated 772 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:59.365952 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb5f0d8-0e98-459d-9c7e-93db9841b8ef None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:59.383278 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77ff096-854a-4868-ae15-144ce6fb0356 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:59.383278 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77ff096-854a-4868-ae15-144ce6fb0356 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:59.383278 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77ff096-854a-4868-ae15-144ce6fb0356 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:59.383922 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 387/776] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:43:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:43:59.389072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2f190e-d977-4935-b9ad-a7f5a1a2169a None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:59.389611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2f190e-d977-4935-b9ad-a7f5a1a2169a None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:59.390101 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2f190e-d977-4935-b9ad-a7f5a1a2169a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:59.399960 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:59.411047 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-ff2f190e-d977-4935-b9ad-a7f5a1a2169a None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:43:59.412542 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ff2f190e-d977-4935-b9ad-a7f5a1a2169a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:43:59.481294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb5f0d8-0e98-459d-9c7e-93db9841b8ef None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:59.489548 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 388/777] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:43:59 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 23 13:43:59.525004 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66db742-35c6-4798-9bde-63dce228f2a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:59.525705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66db742-35c6-4798-9bde-63dce228f2a7 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:59.525984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66db742-35c6-4798-9bde-63dce228f2a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:59.526247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66db742-35c6-4798-9bde-63dce228f2a7 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:59.526908 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66db742-35c6-4798-9bde-63dce228f2a7 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:59.528354 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66db742-35c6-4798-9bde-63dce228f2a7 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:59.539510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-b66db742-35c6-4798-9bde-63dce228f2a7 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 23 13:43:59.539510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:59.539510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:43:59.539510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:43:59.539510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:43:59.539510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:43:59.539510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:43:59.539510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:43:59.539510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 23 13:43:59.539510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:59.539510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:43:59.539510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:59.539510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:43:59.539510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:43:59.539510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:43:59.539510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:43:59.539510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:43:59.540519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:43:59.540519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:43:59.540519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:43:59.540519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:43:59.540519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:43:59.540519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:43:59.540519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:43:59.540519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:43:59.540519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:43:59.540519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:43:59.540519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:43:59.540519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:43:59.540519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:43:59.540519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:43:59.540519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:43:59.540519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:43:59.541332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:43:59.541332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:43:59.541332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:43:59.541332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:43:59.541332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:43:59.541332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:43:59.541332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:43:59.541332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:43:59.541332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:43:59.541332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:43:59.541332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:43:59.541332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:43:59.541332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 23 13:43:59.541332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:43:59.541332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 390/778] 10.222.0.62 () {56 vars in 1129 bytes} [Fri Sep 23 13:43:59 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:43:59.546658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2f190e-d977-4935-b9ad-a7f5a1a2169a None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:59.583400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:59.668947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2f190e-d977-4935-b9ad-a7f5a1a2169a None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:59.704219 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673c0bcf-fd04-4c4a-a37c-1fa8f7297494 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:59.705214 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673c0bcf-fd04-4c4a-a37c-1fa8f7297494 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:59.705699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673c0bcf-fd04-4c4a-a37c-1fa8f7297494 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:59.706167 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673c0bcf-fd04-4c4a-a37c-1fa8f7297494 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:59.707036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673c0bcf-fd04-4c4a-a37c-1fa8f7297494 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:59.708147 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673c0bcf-fd04-4c4a-a37c-1fa8f7297494 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:59.709693 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-673c0bcf-fd04-4c4a-a37c-1fa8f7297494 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:59.714862 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 389/779] 10.222.0.62 () {56 vars in 1124 bytes} [Fri Sep 23 13:43:59 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:59.724113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66db742-35c6-4798-9bde-63dce228f2a7 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:59.849893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:59.857043 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66db742-35c6-4798-9bde-63dce228f2a7 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:43:59.860726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673c0bcf-fd04-4c4a-a37c-1fa8f7297494 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:59.915170 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a603737-4f2e-4b19-b5a6-3a5682821a46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:43:59.915802 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a603737-4f2e-4b19-b5a6-3a5682821a46 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:43:59.916081 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a603737-4f2e-4b19-b5a6-3a5682821a46 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:43:59.916348 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a603737-4f2e-4b19-b5a6-3a5682821a46 None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:43:59.916985 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a603737-4f2e-4b19-b5a6-3a5682821a46 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:43:59.917285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a603737-4f2e-4b19-b5a6-3a5682821a46 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:43:59.918683 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a603737-4f2e-4b19-b5a6-3a5682821a46 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:43:59.925137 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 391/780] 10.222.0.62 () {56 vars in 1124 bytes} [Fri Sep 23 13:43:59 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:43:59.931530 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a603737-4f2e-4b19-b5a6-3a5682821a46 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:43:59.974744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:43:59.995151 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-673c0bcf-fd04-4c4a-a37c-1fa8f7297494 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:00.075848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0bea43-ead8-482f-85e9-72d8bb4c7604 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:00.076506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0bea43-ead8-482f-85e9-72d8bb4c7604 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:00.076792 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0bea43-ead8-482f-85e9-72d8bb4c7604 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:00.077060 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0bea43-ead8-482f-85e9-72d8bb4c7604 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:00.096578 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0bea43-ead8-482f-85e9-72d8bb4c7604 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:44:00.103173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-3c0bea43-ead8-482f-85e9-72d8bb4c7604 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:44:00.103173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:00.103173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:00.103173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:00.103173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:00.103173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:00.103173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:00.103173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:00.103173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:00.103173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:00.103173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:00.103173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:00.103173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:44:00.103173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:44:00.103173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:44:00.103173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:44:00.104142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:44:00.104142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:44:00.104142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:44:00.104142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:44:00.104142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:00.104142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:00.104142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:00.104142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:00.104142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:00.104142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:00.104142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:00.104142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:00.104142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:00.104142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:00.104142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:00.104142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:00.104977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:00.104977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:00.104977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:00.104977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:00.104977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:44:00.104977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:44:00.104977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:44:00.104977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:44:00.104977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:44:00.104977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:44:00.104977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:44:00.104977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:00.104977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 390/781] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:43:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:00.111323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0bea43-ead8-482f-85e9-72d8bb4c7604 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:00.137404 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a603737-4f2e-4b19-b5a6-3a5682821a46 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:00.205668 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed813bfa-5bd4-4e8a-9302-df0f550bd8c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:00.206302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed813bfa-5bd4-4e8a-9302-df0f550bd8c5 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:00.206604 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed813bfa-5bd4-4e8a-9302-df0f550bd8c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:00.206875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed813bfa-5bd4-4e8a-9302-df0f550bd8c5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:00.207649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed813bfa-5bd4-4e8a-9302-df0f550bd8c5 None admin] RBAC: Adding query filter params (interface=public, service_id=a0b923e99e794053b7b9759f150448ff, region_id=RegionOne) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:00.208062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed813bfa-5bd4-4e8a-9302-df0f550bd8c5 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:00.214641 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed813bfa-5bd4-4e8a-9302-df0f550bd8c5 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:00.223336 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 392/782] 10.222.0.62 () {56 vars in 1251 bytes} [Fri Sep 23 13:43:59 2022] GET /identity/v3/endpoints?service_id=a0b923e99e794053b7b9759f150448ff&interface=public®ion_id=RegionOne => generated 194 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:00.224803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488188d2-3fc7-458b-97b4-ab3c49dc66d8 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:00.225100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488188d2-3fc7-458b-97b4-ab3c49dc66d8 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:00.225364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488188d2-3fc7-458b-97b4-ab3c49dc66d8 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:00.230708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 393/783] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:00.230708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488188d2-3fc7-458b-97b4-ab3c49dc66d8 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:00.259027 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0bea43-ead8-482f-85e9-72d8bb4c7604 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:00.270741 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:00.302971 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae9675e-b598-4177-9c42-4b04161769f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:00.303760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae9675e-b598-4177-9c42-4b04161769f1 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:00.304100 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae9675e-b598-4177-9c42-4b04161769f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:00.304582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae9675e-b598-4177-9c42-4b04161769f1 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:00.305778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae9675e-b598-4177-9c42-4b04161769f1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:00.306201 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae9675e-b598-4177-9c42-4b04161769f1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:00.307618 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae9675e-b598-4177-9c42-4b04161769f1 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:00.318429 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 391/784] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:44:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:00.319820 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b9bc9-c921-4c8c-9a80-c48b4fd4c701 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:00.321947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b9bc9-c921-4c8c-9a80-c48b4fd4c701 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:00.321947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b9bc9-c921-4c8c-9a80-c48b4fd4c701 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:00.331125 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:00.345356 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-e78b9bc9-c921-4c8c-9a80-c48b4fd4c701 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:00.347856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e78b9bc9-c921-4c8c-9a80-c48b4fd4c701 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:00.352123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488188d2-3fc7-458b-97b4-ab3c49dc66d8 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:00.392618 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d7a7ac-1ed5-42c7-bbda-87a5b6209e0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:00.394034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d7a7ac-1ed5-42c7-bbda-87a5b6209e0b None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:00.395132 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d7a7ac-1ed5-42c7-bbda-87a5b6209e0b None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:00.395761 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d7a7ac-1ed5-42c7-bbda-87a5b6209e0b None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:00.396764 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d7a7ac-1ed5-42c7-bbda-87a5b6209e0b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:00.398516 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d7a7ac-1ed5-42c7-bbda-87a5b6209e0b None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:00.411309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 394/785] 10.222.0.62 () {56 vars in 1087 bytes} [Fri Sep 23 13:44:00 2022] GET /identity/v3/services => generated 772 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:00.412679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d7a7ac-1ed5-42c7-bbda-87a5b6209e0b None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:00.429954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 392/786] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:00 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:00.550837 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:00.567736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d7a7ac-1ed5-42c7-bbda-87a5b6209e0b None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:00.623136 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395e2bfe-aaf6-424b-a78c-6dbbfb6be6a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:00.623790 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395e2bfe-aaf6-424b-a78c-6dbbfb6be6a7 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:00.624064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395e2bfe-aaf6-424b-a78c-6dbbfb6be6a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:00.624332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395e2bfe-aaf6-424b-a78c-6dbbfb6be6a7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:00.635404 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395e2bfe-aaf6-424b-a78c-6dbbfb6be6a7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:00.636923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395e2bfe-aaf6-424b-a78c-6dbbfb6be6a7 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:00.657984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-395e2bfe-aaf6-424b-a78c-6dbbfb6be6a7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:00.657984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:00.657984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:00.657984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:00.657984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:00.657984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:00.657984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:00.657984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:00.657984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:00.657984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:00.657984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:00.657984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:00.657984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:00.657984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:00.657984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:00.657984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:00.659025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:00.659025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:00.659025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:00.659025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:00.659025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:00.659025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:00.659025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:00.659025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:00.659025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:00.659025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:00.659025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:00.659025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:00.659025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:00.659025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:00.659025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:00.659025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:00.659025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:00.659896 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:00.659896 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:00.659896 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:00.663603 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 395/787] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:00.669708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78b9bc9-c921-4c8c-9a80-c48b4fd4c701 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:00.704258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395e2bfe-aaf6-424b-a78c-6dbbfb6be6a7 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:00.807092 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78b9bc9-c921-4c8c-9a80-c48b4fd4c701 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:00.850814 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e913ce25-a866-489f-bedc-2abee2f14284 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:00.851550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e913ce25-a866-489f-bedc-2abee2f14284 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:00.851823 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e913ce25-a866-489f-bedc-2abee2f14284 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:00.852087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e913ce25-a866-489f-bedc-2abee2f14284 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:00.852744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e913ce25-a866-489f-bedc-2abee2f14284 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:00.853033 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e913ce25-a866-489f-bedc-2abee2f14284 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:00.854449 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e913ce25-a866-489f-bedc-2abee2f14284 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:00.856163 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:00.862800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395e2bfe-aaf6-424b-a78c-6dbbfb6be6a7 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:00.868388 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 393/788] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:00.873763 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e913ce25-a866-489f-bedc-2abee2f14284 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:00.908639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2133a3-b073-44ba-b43a-f0a799b4e86e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:00.911137 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2133a3-b073-44ba-b43a-f0a799b4e86e None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:00.911861 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2133a3-b073-44ba-b43a-f0a799b4e86e None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:00.912460 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2133a3-b073-44ba-b43a-f0a799b4e86e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:00.921884 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:00.928807 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2133a3-b073-44ba-b43a-f0a799b4e86e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:44:00.935278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-ae2133a3-b073-44ba-b43a-f0a799b4e86e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:44:00.935278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:00.935278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:00.935278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:00.935278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:00.935278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:00.935278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:00.935278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:00.935278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:00.935278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:00.935278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:00.935278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:00.935278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:44:00.935278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:44:00.935278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:44:00.935278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:44:00.940243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:44:00.940243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:44:00.940243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:44:00.940243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:44:00.940243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:00.940243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:00.940243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:00.940243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:00.940243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:00.940243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:00.940243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:00.940243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:00.940243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:00.940243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:00.940243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:00.940243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:00.941105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:00.941105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:00.941105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:00.941105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:00.941105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:44:00.941105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:44:00.941105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:44:00.941105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:44:00.941105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:44:00.941105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:44:00.941105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:44:00.941105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:00.941105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 396/789] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:44:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:00.947696 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2133a3-b073-44ba-b43a-f0a799b4e86e None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:00.996877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e913ce25-a866-489f-bedc-2abee2f14284 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:01.033050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64741ad9-79e6-48c7-b992-20e5244ca014 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:01.033684 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64741ad9-79e6-48c7-b992-20e5244ca014 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:01.033963 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64741ad9-79e6-48c7-b992-20e5244ca014 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:01.034232 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64741ad9-79e6-48c7-b992-20e5244ca014 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:01.043280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64741ad9-79e6-48c7-b992-20e5244ca014 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:01.044810 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64741ad9-79e6-48c7-b992-20e5244ca014 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:01.058553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-64741ad9-79e6-48c7-b992-20e5244ca014 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 23 13:44:01.058553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:01.058553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:01.058553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:01.058553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:01.058553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:01.058553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:01.058553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:01.058553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:01.058553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:01.058553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:01.058553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:01.058553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:44:01.058553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:44:01.058553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:44:01.058553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:44:01.059501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:01.059501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:01.059501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:44:01.059501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:01.059501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:44:01.059501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:01.059501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:01.059501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:01.059501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:01.059501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:01.059501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:01.059501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:01.059501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:01.059501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:01.059501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:01.059501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:01.059501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:01.059501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:01.060451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:01.060451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:01.060451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:44:01.060451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:44:01.060451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:44:01.060451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:44:01.060451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:44:01.060451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:44:01.060451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 23 13:44:01.060451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:01.060451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 394/790] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:44:00 2022] GET /identity/v3/users/swift => generated 83 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:01.063769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64741ad9-79e6-48c7-b992-20e5244ca014 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:01.170798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:01.259774 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64741ad9-79e6-48c7-b992-20e5244ca014 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:01.300194 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2133a3-b073-44ba-b43a-f0a799b4e86e None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:01.323960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab00b4b-5ef6-42b0-b4bb-68e9cb926366 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:01.324494 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab00b4b-5ef6-42b0-b4bb-68e9cb926366 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:01.324771 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab00b4b-5ef6-42b0-b4bb-68e9cb926366 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:01.325041 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab00b4b-5ef6-42b0-b4bb-68e9cb926366 None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:01.325774 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab00b4b-5ef6-42b0-b4bb-68e9cb926366 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:01.326077 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab00b4b-5ef6-42b0-b4bb-68e9cb926366 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:01.327551 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab00b4b-5ef6-42b0-b4bb-68e9cb926366 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:01.354818 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 395/791] 10.222.0.62 () {56 vars in 1135 bytes} [Fri Sep 23 13:44:01 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:01.356222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab00b4b-5ef6-42b0-b4bb-68e9cb926366 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:01.371655 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86dc4c44-7334-41cc-9255-8d77969dfbd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:01.372282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86dc4c44-7334-41cc-9255-8d77969dfbd4 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:01.372559 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86dc4c44-7334-41cc-9255-8d77969dfbd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:01.372827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86dc4c44-7334-41cc-9255-8d77969dfbd4 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:01.373498 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86dc4c44-7334-41cc-9255-8d77969dfbd4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:01.373797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86dc4c44-7334-41cc-9255-8d77969dfbd4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:01.375202 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86dc4c44-7334-41cc-9255-8d77969dfbd4 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:01.390294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 397/792] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:44:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 439 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:01.390294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f79a0d-8900-494c-9aec-ae523d75cca1 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:01.390294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f79a0d-8900-494c-9aec-ae523d75cca1 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:01.390294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f79a0d-8900-494c-9aec-ae523d75cca1 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:01.394066 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 398/793] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:01 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:01.395456 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f79a0d-8900-494c-9aec-ae523d75cca1 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:01.474158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:01.480520 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab00b4b-5ef6-42b0-b4bb-68e9cb926366 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:01.495396 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:01.519946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d45cc54-5013-4551-a48c-10958e45ed63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:01.520692 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d45cc54-5013-4551-a48c-10958e45ed63 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:01.520995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d45cc54-5013-4551-a48c-10958e45ed63 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:01.521339 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d45cc54-5013-4551-a48c-10958e45ed63 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:01.532482 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f79a0d-8900-494c-9aec-ae523d75cca1 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:01.533038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d45cc54-5013-4551-a48c-10958e45ed63 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:01.534915 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d45cc54-5013-4551-a48c-10958e45ed63 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:01.551472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-8d45cc54-5013-4551-a48c-10958e45ed63 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:01.551472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:01.551472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:01.551472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:01.551472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:01.551472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:01.551472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:01.551472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:01.551472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:01.551472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:01.551472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:01.551472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:01.551472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:01.551472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:01.551472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:01.551472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:01.552478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:01.552478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:01.552478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:01.552478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:01.552478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:01.552478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:01.552478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:01.552478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:01.552478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:01.552478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:01.552478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:01.552478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:01.552478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:01.552478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:01.552478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:01.552478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:01.552478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:01.553383 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:01.553383 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:01.553383 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:01.557566 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 396/794] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:44:01.559000 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc827dc5-4538-4c27-be9f-f5804126a7d2 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:01.559348 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc827dc5-4538-4c27-be9f-f5804126a7d2 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:01.559687 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc827dc5-4538-4c27-be9f-f5804126a7d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:01.580082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ea4920-6001-44ca-9e97-82c83b3af291 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:01.580755 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ea4920-6001-44ca-9e97-82c83b3af291 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:01.581102 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ea4920-6001-44ca-9e97-82c83b3af291 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:01.581559 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ea4920-6001-44ca-9e97-82c83b3af291 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:01.583212 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-fc827dc5-4538-4c27-be9f-f5804126a7d2 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:01.584613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-fc827dc5-4538-4c27-be9f-f5804126a7d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:01.595859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ea4920-6001-44ca-9e97-82c83b3af291 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:01.598314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ea4920-6001-44ca-9e97-82c83b3af291 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:01.619027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-48ea4920-6001-44ca-9e97-82c83b3af291 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:01.619027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:01.619027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:01.619027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:01.619027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:01.619027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:01.619027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:01.619027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:01.619027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:01.619027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:01.619027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:01.619027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:01.619027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:01.619027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:01.619027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:01.619027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:01.620048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:01.620048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:01.620048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:01.620048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:01.620048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:01.620048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:01.620048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:01.620048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:01.620048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:01.620048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:01.620048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:01.620048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:01.620048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:01.620048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:01.620048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:01.620048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:01.620048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:01.620940 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:01.620940 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:01.620940 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:01.622470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 399/795] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:01.623525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ea4920-6001-44ca-9e97-82c83b3af291 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:01.670589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 397/796] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:01 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:01.671840 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc827dc5-4538-4c27-be9f-f5804126a7d2 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:01.770908 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ea4920-6001-44ca-9e97-82c83b3af291 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:01.790628 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:01.832871 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b831ad8d-a521-4c21-abe8-669560112abf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:01.833478 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b831ad8d-a521-4c21-abe8-669560112abf None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:01.833762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b831ad8d-a521-4c21-abe8-669560112abf None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:01.834034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b831ad8d-a521-4c21-abe8-669560112abf None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:01.834702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b831ad8d-a521-4c21-abe8-669560112abf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:01.834999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b831ad8d-a521-4c21-abe8-669560112abf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:01.836355 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b831ad8d-a521-4c21-abe8-669560112abf None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:01.860136 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 400/797] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:01.860932 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8752020a-e4bb-4730-a4b1-50ddde21506e None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:01.861210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8752020a-e4bb-4730-a4b1-50ddde21506e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:01.861477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8752020a-e4bb-4730-a4b1-50ddde21506e None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:01.862930 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 401/798] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:01.866565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8752020a-e4bb-4730-a4b1-50ddde21506e None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:01.872899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:01.880278 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc827dc5-4538-4c27-be9f-f5804126a7d2 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:01.918384 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdf7da8-e59d-44c3-8f89-dab49e16cec0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:01.918991 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdf7da8-e59d-44c3-8f89-dab49e16cec0 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:01.919324 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdf7da8-e59d-44c3-8f89-dab49e16cec0 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:01.919596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdf7da8-e59d-44c3-8f89-dab49e16cec0 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:01.920295 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdf7da8-e59d-44c3-8f89-dab49e16cec0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:01.920595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdf7da8-e59d-44c3-8f89-dab49e16cec0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:01.921944 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdf7da8-e59d-44c3-8f89-dab49e16cec0 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:01.935007 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 398/799] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:01.936181 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12477d04-ae3a-4ee4-9436-05e68da45b03 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:01.936462 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12477d04-ae3a-4ee4-9436-05e68da45b03 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:01.936735 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12477d04-ae3a-4ee4-9436-05e68da45b03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:01.959386 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-12477d04-ae3a-4ee4-9436-05e68da45b03 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:01.961511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-12477d04-ae3a-4ee4-9436-05e68da45b03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:02.000048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8752020a-e4bb-4730-a4b1-50ddde21506e None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:02.036948 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 399/800] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:01 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:02.037786 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12477d04-ae3a-4ee4-9436-05e68da45b03 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:02.038631 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cc1a0d-f81b-4dc2-b357-31327f9cdbc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:02.039516 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cc1a0d-f81b-4dc2-b357-31327f9cdbc1 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:02.040047 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cc1a0d-f81b-4dc2-b357-31327f9cdbc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:02.040384 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cc1a0d-f81b-4dc2-b357-31327f9cdbc1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:02.047329 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cc1a0d-f81b-4dc2-b357-31327f9cdbc1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:02.048850 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cc1a0d-f81b-4dc2-b357-31327f9cdbc1 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:02.061601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-c5cc1a0d-f81b-4dc2-b357-31327f9cdbc1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:02.061601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:02.061601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:02.061601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:02.061601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:02.061601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:02.061601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:02.061601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:02.061601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:02.061601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:02.061601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:02.061601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:02.061601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:44:02.061601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:44:02.061601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:44:02.062538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:44:02.062538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:02.062538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:02.062538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:02.062538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:02.062538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:02.062538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:02.062538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:02.062538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:02.062538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:02.062538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:02.062538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:02.062538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:02.062538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:02.062538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:02.062538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:02.062538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:02.063520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:44:02.063520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:44:02.063520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:44:02.063520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:44:02.063520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:44:02.063520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:44:02.063520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:02.063520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:02.067587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 402/801] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:44:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:02.068456 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cc1a0d-f81b-4dc2-b357-31327f9cdbc1 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:02.106870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:02.148056 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:02.168527 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12477d04-ae3a-4ee4-9436-05e68da45b03 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:02.177173 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cc1a0d-f81b-4dc2-b357-31327f9cdbc1 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:02.214656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa2a7d4-57f4-4800-b2d9-34e91b28439c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:02.215302 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa2a7d4-57f4-4800-b2d9-34e91b28439c None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:02.215581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa2a7d4-57f4-4800-b2d9-34e91b28439c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:02.215847 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa2a7d4-57f4-4800-b2d9-34e91b28439c None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:02.227996 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa2a7d4-57f4-4800-b2d9-34e91b28439c None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:02.228146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4f2511-94d9-4a80-a9ca-109f778f29e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:02.228710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4f2511-94d9-4a80-a9ca-109f778f29e8 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:02.228989 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4f2511-94d9-4a80-a9ca-109f778f29e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:02.229259 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4f2511-94d9-4a80-a9ca-109f778f29e8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:02.230756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa2a7d4-57f4-4800-b2d9-34e91b28439c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:02.241700 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4f2511-94d9-4a80-a9ca-109f778f29e8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:44:02.245856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-3fa2a7d4-57f4-4800-b2d9-34e91b28439c None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 23 13:44:02.245856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:02.245856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:02.245856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:02.245856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:02.245856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:02.245856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:02.245856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:02.245856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:02.245856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:02.245856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:02.245856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:02.245856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:44:02.245856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:44:02.245856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:44:02.245856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:44:02.246828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:02.246828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:02.246828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:44:02.246828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:02.246828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:44:02.246828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:02.246828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:02.246828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:02.246828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:02.246828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:02.246828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:02.246828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:02.246828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:02.246828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:02.246828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:02.246828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:02.246828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:02.246828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:02.247758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:02.247758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:02.247758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:44:02.247758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:44:02.247758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:44:02.247758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:44:02.247758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:44:02.247758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:44:02.247758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 23 13:44:02.247758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:02.248282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-ee4f2511-94d9-4a80-a9ca-109f778f29e8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:44:02.248282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:02.248282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:02.248282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:02.248282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:02.248282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:02.248282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:02.248282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:02.248282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:02.248282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:02.248282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:02.248282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:02.248282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:44:02.248282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:44:02.248282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:44:02.248282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:44:02.249100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:44:02.249100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:44:02.249100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:44:02.249100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:44:02.249100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:02.249100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:02.249100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:02.249100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:02.249100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:02.249100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:02.249100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:02.249100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:02.249100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:02.249100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:02.249100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:02.249100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:02.249935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:02.249935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:02.249935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:02.249935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:02.249935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:44:02.249935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:44:02.249935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:44:02.249935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:44:02.249935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:44:02.249935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:44:02.249935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:44:02.249935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:02.252877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 400/802] 10.222.0.62 () {56 vars in 1117 bytes} [Fri Sep 23 13:44:02 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:02.253444 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 403/803] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:44:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:02.254885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4f2511-94d9-4a80-a9ca-109f778f29e8 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:02.255177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa2a7d4-57f4-4800-b2d9-34e91b28439c None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:02.362962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:02.370859 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa2a7d4-57f4-4800-b2d9-34e91b28439c None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:02.372957 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4f2511-94d9-4a80-a9ca-109f778f29e8 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:02.406966 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb236287-5344-40f5-b288-188d73111472 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:02.407569 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb236287-5344-40f5-b288-188d73111472 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:02.407848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb236287-5344-40f5-b288-188d73111472 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:02.408115 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb236287-5344-40f5-b288-188d73111472 None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:02.408425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5f890f-6ed2-47b5-b01a-f4b93f8cd9a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:02.408859 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb236287-5344-40f5-b288-188d73111472 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:02.409032 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5f890f-6ed2-47b5-b01a-f4b93f8cd9a3 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:02.409134 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb236287-5344-40f5-b288-188d73111472 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:02.409308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5f890f-6ed2-47b5-b01a-f4b93f8cd9a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:02.409548 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5f890f-6ed2-47b5-b01a-f4b93f8cd9a3 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:02.410853 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5f890f-6ed2-47b5-b01a-f4b93f8cd9a3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:02.411156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5f890f-6ed2-47b5-b01a-f4b93f8cd9a3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:02.411377 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb236287-5344-40f5-b288-188d73111472 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:02.412487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5f890f-6ed2-47b5-b01a-f4b93f8cd9a3 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:02.425175 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 404/804] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:44:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:02.425175 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5f890f-6ed2-47b5-b01a-f4b93f8cd9a3 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:02.429342 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 401/805] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:44:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:02.434226 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb236287-5344-40f5-b288-188d73111472 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:02.437409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:02.533239 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5f890f-6ed2-47b5-b01a-f4b93f8cd9a3 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:02.537136 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb236287-5344-40f5-b288-188d73111472 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:02.570723 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9700c5-0f43-4e2a-b808-b142dd66d2bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:02.571331 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9700c5-0f43-4e2a-b808-b142dd66d2bf None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:02.571601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9700c5-0f43-4e2a-b808-b142dd66d2bf None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:02.571862 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9700c5-0f43-4e2a-b808-b142dd66d2bf None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:02.572580 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9700c5-0f43-4e2a-b808-b142dd66d2bf None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:02.572878 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9700c5-0f43-4e2a-b808-b142dd66d2bf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:02.573191 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928efd1b-2526-4281-a71d-516822fbec33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:02.573812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928efd1b-2526-4281-a71d-516822fbec33 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:02.574086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928efd1b-2526-4281-a71d-516822fbec33 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:02.574279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9700c5-0f43-4e2a-b808-b142dd66d2bf None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:02.574395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928efd1b-2526-4281-a71d-516822fbec33 None admin] PATH_INFO: `/v3/services/network` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:02.574940 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928efd1b-2526-4281-a71d-516822fbec33 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:02.576263 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928efd1b-2526-4281-a71d-516822fbec33 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:02.585805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-928efd1b-2526-4281-a71d-516822fbec33 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 23 13:44:02.585805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:02.585805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:44:02.585805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:44:02.585805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:44:02.585805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:44:02.585805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:44:02.585805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:02.585805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 23 13:44:02.585805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:02.585805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:02.585805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:02.585805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:02.585805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:02.585805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:02.585805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:02.585805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:02.587037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:02.587037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:02.587037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:02.587037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:02.587037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:02.587037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:02.587037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:44:02.587037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:44:02.587037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:44:02.587037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:44:02.587037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:02.587037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:02.587037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:02.587037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:02.587037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:02.587037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:02.587853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:02.587853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:02.587853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:02.587853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:02.587853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:02.587853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:02.587853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:02.587853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:02.587853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:02.587853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:02.587853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:44:02.587853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:02.587853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 23 13:44:02.587853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:02.591790 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 402/806] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:44:02 2022] GET /identity/v3/services/network => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:02.592666 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928efd1b-2526-4281-a71d-516822fbec33 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:02.592770 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 405/807] 10.222.0.62 () {56 vars in 1149 bytes} [Fri Sep 23 13:44:02 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:02.593337 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9700c5-0f43-4e2a-b808-b142dd66d2bf None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:02.609043 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:02.672165 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:02.698364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9700c5-0f43-4e2a-b808-b142dd66d2bf None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:02.708948 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928efd1b-2526-4281-a71d-516822fbec33 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:02.734913 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde4d547-509b-4131-9d70-8cded0d0f2b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:02.735573 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde4d547-509b-4131-9d70-8cded0d0f2b0 None admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:02.735854 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde4d547-509b-4131-9d70-8cded0d0f2b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:02.736124 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde4d547-509b-4131-9d70-8cded0d0f2b0 None admin] PATH_INFO: `/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/4b9269509f1d454eab7a3008c2ba0039/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:02.738075 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde4d547-509b-4131-9d70-8cded0d0f2b0 None admin] RBAC: Authorizing `identity:create_grant(project_id=2a19fd4315ad4a6eb349dfe8cfa66c23, user_id=4b9269509f1d454eab7a3008c2ba0039, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:02.739715 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde4d547-509b-4131-9d70-8cded0d0f2b0 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:02.745665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5a7791-85b9-46b5-aea2-b09d7002b680 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:02.746296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5a7791-85b9-46b5-aea2-b09d7002b680 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:02.746601 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5a7791-85b9-46b5-aea2-b09d7002b680 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:02.746870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b5a7791-85b9-46b5-aea2-b09d7002b680 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:02.757770 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5a7791-85b9-46b5-aea2-b09d7002b680 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:02.759544 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b5a7791-85b9-46b5-aea2-b09d7002b680 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:02.773869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-7b5a7791-85b9-46b5-aea2-b09d7002b680 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:02.773869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:02.773869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:02.773869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:02.773869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:02.773869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:02.773869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:02.773869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:02.773869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:02.773869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:02.773869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:02.773869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:02.773869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:02.773869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:02.773869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:02.773869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:02.774883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:02.774883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:02.774883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:02.774883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:02.774883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:02.774883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:02.774883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:02.774883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:02.774883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:02.774883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:02.774883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:02.774883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:02.774883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:02.774883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:02.774883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:02.774883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:02.774883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:02.775766 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:02.775766 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:02.775766 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:02.775766 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 403/808] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:02.775995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5a7791-85b9-46b5-aea2-b09d7002b680 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:02.818710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 406/809] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:44:02 2022] PUT /identity/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/4b9269509f1d454eab7a3008c2ba0039/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:44:02.819650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde4d547-509b-4131-9d70-8cded0d0f2b0 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:02.908590 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:02.920391 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b5a7791-85b9-46b5-aea2-b09d7002b680 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:02.946798 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde4d547-509b-4131-9d70-8cded0d0f2b0 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:02.948996 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:02.973265 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbfeab6-fc2a-4678-94bc-7f1b98e55c93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:02.974114 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbfeab6-fc2a-4678-94bc-7f1b98e55c93 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:02.974432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbfeab6-fc2a-4678-94bc-7f1b98e55c93 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:02.975198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbfeab6-fc2a-4678-94bc-7f1b98e55c93 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:02.975907 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bbfeab6-fc2a-4678-94bc-7f1b98e55c93 None admin] RBAC: Adding query filter params (name=network) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:02.976213 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bbfeab6-fc2a-4678-94bc-7f1b98e55c93 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:02.978399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bbfeab6-fc2a-4678-94bc-7f1b98e55c93 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:02.984663 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 404/810] 10.222.0.62 () {56 vars in 1112 bytes} [Fri Sep 23 13:44:02 2022] GET /identity/v3/services?name=network => generated 124 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:02.985585 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbfeab6-fc2a-4678-94bc-7f1b98e55c93 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:02.997530 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50caf9c1-3608-451e-84b9-bf251e387c69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:02.999452 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50caf9c1-3608-451e-84b9-bf251e387c69 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:03.000017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50caf9c1-3608-451e-84b9-bf251e387c69 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:03.000490 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50caf9c1-3608-451e-84b9-bf251e387c69 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:03.015909 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50caf9c1-3608-451e-84b9-bf251e387c69 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:03.018547 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50caf9c1-3608-451e-84b9-bf251e387c69 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:03.040099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-50caf9c1-3608-451e-84b9-bf251e387c69 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:03.040099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:03.040099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:03.040099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:03.040099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:03.040099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:03.040099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:03.040099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:03.040099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:03.040099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:03.040099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:03.040099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:03.040099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:03.040099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:03.040099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:03.040099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:03.041087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:03.041087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:03.041087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:03.041087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:03.041087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:03.041087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:03.041087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:03.041087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:03.041087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:03.041087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:03.041087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:03.041087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:03.041087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:03.041087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:03.041087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:03.041087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:03.041087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:03.041962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:03.041962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:03.041962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:03.044173 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 407/811] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:03.045705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50caf9c1-3608-451e-84b9-bf251e387c69 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:03.115387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbfeab6-fc2a-4678-94bc-7f1b98e55c93 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:03.155391 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb65606-9761-40a9-bb5f-34493e373cb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:03.156003 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb65606-9761-40a9-bb5f-34493e373cb5 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:03.156543 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb65606-9761-40a9-bb5f-34493e373cb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:03.156855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb65606-9761-40a9-bb5f-34493e373cb5 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:03.157983 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb65606-9761-40a9-bb5f-34493e373cb5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:03.158462 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb65606-9761-40a9-bb5f-34493e373cb5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:03.159824 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb65606-9761-40a9-bb5f-34493e373cb5 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:03.160715 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50caf9c1-3608-451e-84b9-bf251e387c69 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:03.180398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 405/812] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:03.180398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb65606-9761-40a9-bb5f-34493e373cb5 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:03.192219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:03.208658 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705ead4d-90ac-42cf-b6d8-f3e76a941c86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:03.209199 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705ead4d-90ac-42cf-b6d8-f3e76a941c86 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:03.209516 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705ead4d-90ac-42cf-b6d8-f3e76a941c86 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:03.209814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705ead4d-90ac-42cf-b6d8-f3e76a941c86 None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:03.210492 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705ead4d-90ac-42cf-b6d8-f3e76a941c86 None admin] RBAC: Adding query filter params (type=network) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:03.210790 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705ead4d-90ac-42cf-b6d8-f3e76a941c86 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:03.212149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705ead4d-90ac-42cf-b6d8-f3e76a941c86 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:03.218088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 408/813] 10.222.0.62 () {56 vars in 1112 bytes} [Fri Sep 23 13:44:03 2022] GET /identity/v3/services?type=network => generated 350 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:03.219417 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e494c28-ec25-435c-b69b-b4057235231c None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:03.219701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e494c28-ec25-435c-b69b-b4057235231c None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:03.219959 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e494c28-ec25-435c-b69b-b4057235231c None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:03.221296 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 409/814] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:03 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:03.223994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e494c28-ec25-435c-b69b-b4057235231c None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:03.238228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:03.301154 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb65606-9761-40a9-bb5f-34493e373cb5 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:03.336806 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18661bc5-ad40-404f-b54f-dd3a75f4d876 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:03.337572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18661bc5-ad40-404f-b54f-dd3a75f4d876 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:03.337849 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18661bc5-ad40-404f-b54f-dd3a75f4d876 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:03.338433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18661bc5-ad40-404f-b54f-dd3a75f4d876 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:03.339077 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18661bc5-ad40-404f-b54f-dd3a75f4d876 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:03.339379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18661bc5-ad40-404f-b54f-dd3a75f4d876 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:03.341167 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18661bc5-ad40-404f-b54f-dd3a75f4d876 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:03.353449 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 406/815] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:03.354346 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18661bc5-ad40-404f-b54f-dd3a75f4d876 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:03.359656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e494c28-ec25-435c-b69b-b4057235231c None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:03.398378 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0857250-0839-4442-aaa6-c8b53b5d946b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:03.399364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0857250-0839-4442-aaa6-c8b53b5d946b None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:03.399644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0857250-0839-4442-aaa6-c8b53b5d946b None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:03.399914 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0857250-0839-4442-aaa6-c8b53b5d946b None admin] PATH_INFO: `/v3/users/nova` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:03.408000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0857250-0839-4442-aaa6-c8b53b5d946b None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:03.409432 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0857250-0839-4442-aaa6-c8b53b5d946b None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:03.421019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-d0857250-0839-4442-aaa6-c8b53b5d946b None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 23 13:44:03.421019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:03.421019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:03.421019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:03.421019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:03.421019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:03.421019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:03.421019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:03.421019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:03.421019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:03.421019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:03.421019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:03.421019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:44:03.421019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:44:03.421019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:44:03.421019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:44:03.421988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:03.421988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:03.421988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:44:03.421988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:03.421988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:44:03.421988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:03.421988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:03.421988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:03.421988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:03.421988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:03.421988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:03.421988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:03.421988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:03.421988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:03.421988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:03.421988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:03.421988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:03.421988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:03.428750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:03.428750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:03.428750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:44:03.428750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:44:03.428750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:44:03.428750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:44:03.428750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:44:03.428750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:44:03.428750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 23 13:44:03.428750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:03.428750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 410/816] 10.222.0.62 () {56 vars in 1093 bytes} [Fri Sep 23 13:44:03 2022] GET /identity/v3/users/nova => generated 82 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:03.428750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f8701c-37bf-4375-a244-73d50fbe6267 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:03.428750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f8701c-37bf-4375-a244-73d50fbe6267 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:03.429464 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f8701c-37bf-4375-a244-73d50fbe6267 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:03.447604 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-29f8701c-37bf-4375-a244-73d50fbe6267 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:03.448777 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-29f8701c-37bf-4375-a244-73d50fbe6267 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:03.465144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18661bc5-ad40-404f-b54f-dd3a75f4d876 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:03.468981 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:03.491672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:03.502410 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14727c11-907e-4e18-b8dc-c623f8541c5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:03.503018 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14727c11-907e-4e18-b8dc-c623f8541c5e None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:03.503323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14727c11-907e-4e18-b8dc-c623f8541c5e None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:03.503572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14727c11-907e-4e18-b8dc-c623f8541c5e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:03.504787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14727c11-907e-4e18-b8dc-c623f8541c5e None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:03.506138 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14727c11-907e-4e18-b8dc-c623f8541c5e None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:03.514708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 411/817] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:03 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:03.527478 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f8701c-37bf-4375-a244-73d50fbe6267 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:03.631560 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 407/818] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 13:44:03 2022] POST /identity/v3/endpoints => generated 337 bytes in 277 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:44:03.633190 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14727c11-907e-4e18-b8dc-c623f8541c5e None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:03.671624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f8701c-37bf-4375-a244-73d50fbe6267 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:03.827340 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba275ce-1fd3-44eb-a893-4408763aab84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:03.828003 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba275ce-1fd3-44eb-a893-4408763aab84 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:03.828360 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba275ce-1fd3-44eb-a893-4408763aab84 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:03.828631 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba275ce-1fd3-44eb-a893-4408763aab84 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:03.837478 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba275ce-1fd3-44eb-a893-4408763aab84 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:03.839106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba275ce-1fd3-44eb-a893-4408763aab84 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:03.839542 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14727c11-907e-4e18-b8dc-c623f8541c5e None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:03.858147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-fba275ce-1fd3-44eb-a893-4408763aab84 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:03.858147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:03.858147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:03.858147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:03.858147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:03.858147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:03.858147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:03.858147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:03.858147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:03.858147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:03.858147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:03.858147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:03.858147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:44:03.858147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:44:03.858147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:44:03.859103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:44:03.859103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:03.859103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:03.859103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:03.859103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:03.859103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:03.859103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:03.859103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:03.859103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:03.859103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:03.859103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:03.859103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:03.859103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:03.859103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:03.859103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:03.859103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:03.859103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:03.860064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:44:03.860064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:44:03.860064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:44:03.860064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:44:03.860064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:44:03.860064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:44:03.860064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:03.860064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:03.860876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 412/819] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:44:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 345 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:03.868573 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba275ce-1fd3-44eb-a893-4408763aab84 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:03.886253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5f9754-e959-4c3c-9a16-9886ba1b4e60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:03.887023 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5f9754-e959-4c3c-9a16-9886ba1b4e60 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:03.887999 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5f9754-e959-4c3c-9a16-9886ba1b4e60 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:03.887999 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba5f9754-e959-4c3c-9a16-9886ba1b4e60 None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:03.888621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5f9754-e959-4c3c-9a16-9886ba1b4e60 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:03.889089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5f9754-e959-4c3c-9a16-9886ba1b4e60 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:03.891470 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba5f9754-e959-4c3c-9a16-9886ba1b4e60 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:03.920438 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:03.926248 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 408/820] 10.222.0.62 () {56 vars in 1133 bytes} [Fri Sep 23 13:44:03 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:03.928292 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5f9754-e959-4c3c-9a16-9886ba1b4e60 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:03.966709 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:04.017049 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba275ce-1fd3-44eb-a893-4408763aab84 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:04.053079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88373b0-4a75-4ba2-86ce-676e3a2bbd80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:04.053733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88373b0-4a75-4ba2-86ce-676e3a2bbd80 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:04.053983 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88373b0-4a75-4ba2-86ce-676e3a2bbd80 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:04.054252 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88373b0-4a75-4ba2-86ce-676e3a2bbd80 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:04.054857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88373b0-4a75-4ba2-86ce-676e3a2bbd80 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:04.056284 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88373b0-4a75-4ba2-86ce-676e3a2bbd80 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:04.065956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-a88373b0-4a75-4ba2-86ce-676e3a2bbd80 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 23 13:44:04.065956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:04.065956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:44:04.065956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:44:04.065956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:44:04.065956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:44:04.065956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:44:04.065956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:04.065956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 23 13:44:04.065956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:04.065956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:04.065956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:04.065956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:04.065956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:04.065956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:04.065956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:04.065956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:04.067718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:04.067718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:04.067718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:04.067718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:04.067718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:04.067718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:04.067718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:44:04.067718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:44:04.067718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:44:04.067718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:44:04.067718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:04.067718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:04.067718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:04.067718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:04.067718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:04.067718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:04.068527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:04.068527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:04.068527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:04.068527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:04.068527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:04.068527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:04.068527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:04.068527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:04.068527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:04.068527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:04.068527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:44:04.068527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:04.068527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 23 13:44:04.068527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:04.068527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 413/821] 10.222.0.62 () {56 vars in 1129 bytes} [Fri Sep 23 13:44:03 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:04.070153 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88373b0-4a75-4ba2-86ce-676e3a2bbd80 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:04.081636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba5f9754-e959-4c3c-9a16-9886ba1b4e60 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:04.133144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6788e87-80c0-48de-9e37-2fba47f4969b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:04.136333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6788e87-80c0-48de-9e37-2fba47f4969b None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:04.136333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6788e87-80c0-48de-9e37-2fba47f4969b None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:04.137547 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6788e87-80c0-48de-9e37-2fba47f4969b None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:04.138269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6788e87-80c0-48de-9e37-2fba47f4969b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:04.138603 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6788e87-80c0-48de-9e37-2fba47f4969b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:04.143240 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6788e87-80c0-48de-9e37-2fba47f4969b None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:04.164448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 409/822] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:44:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:04.165764 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6788e87-80c0-48de-9e37-2fba47f4969b None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:04.180973 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:04.187248 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88373b0-4a75-4ba2-86ce-676e3a2bbd80 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:04.236971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498f7a6f-0fd4-4841-89ce-69484764d3e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:04.237646 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498f7a6f-0fd4-4841-89ce-69484764d3e9 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:04.237845 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498f7a6f-0fd4-4841-89ce-69484764d3e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:04.238167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498f7a6f-0fd4-4841-89ce-69484764d3e9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:04.274680 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:04.297678 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6788e87-80c0-48de-9e37-2fba47f4969b None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:04.308072 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498f7a6f-0fd4-4841-89ce-69484764d3e9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:04.309509 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498f7a6f-0fd4-4841-89ce-69484764d3e9 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:04.334940 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8b00a3-0f3e-4d61-9cf9-3050211cbc44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:04.335642 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8b00a3-0f3e-4d61-9cf9-3050211cbc44 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:04.336018 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8b00a3-0f3e-4d61-9cf9-3050211cbc44 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:04.336390 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8b00a3-0f3e-4d61-9cf9-3050211cbc44 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:04.337281 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb8b00a3-0f3e-4d61-9cf9-3050211cbc44 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:04.338002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb8b00a3-0f3e-4d61-9cf9-3050211cbc44 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:04.339499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb8b00a3-0f3e-4d61-9cf9-3050211cbc44 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:04.348676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 410/823] 10.222.0.62 () {56 vars in 1124 bytes} [Fri Sep 23 13:44:04 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:04.348676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8b00a3-0f3e-4d61-9cf9-3050211cbc44 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:04.412523 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:04.417165 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 414/824] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:44:04 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 23 13:44:04.417804 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498f7a6f-0fd4-4841-89ce-69484764d3e9 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:04.457681 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8b00a3-0f3e-4d61-9cf9-3050211cbc44 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:04.493354 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf13af48-7e1f-447c-98c1-68f4a19f7d3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:04.494354 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf13af48-7e1f-447c-98c1-68f4a19f7d3c None admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:04.494835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf13af48-7e1f-447c-98c1-68f4a19f7d3c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:04.495352 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf13af48-7e1f-447c-98c1-68f4a19f7d3c None admin] PATH_INFO: `/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/b471d768cd814ff8b4f6bd35d21e24a8/roles/809612fc6d62442cbcc179b5c4123465` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:04.509123 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:04.512893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf13af48-7e1f-447c-98c1-68f4a19f7d3c None admin] RBAC: Authorizing `identity:create_grant(project_id=2a19fd4315ad4a6eb349dfe8cfa66c23, user_id=b471d768cd814ff8b4f6bd35d21e24a8, role_id=809612fc6d62442cbcc179b5c4123465)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:04.514559 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf13af48-7e1f-447c-98c1-68f4a19f7d3c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:04.521241 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498f7a6f-0fd4-4841-89ce-69484764d3e9 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:04.555203 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89a075f-9f70-45db-87f1-c72a96e42a9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:04.555803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89a075f-9f70-45db-87f1-c72a96e42a9c None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:04.556079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89a075f-9f70-45db-87f1-c72a96e42a9c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:04.556348 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89a075f-9f70-45db-87f1-c72a96e42a9c None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:04.556972 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89a075f-9f70-45db-87f1-c72a96e42a9c None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:04.557273 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89a075f-9f70-45db-87f1-c72a96e42a9c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:04.558780 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89a075f-9f70-45db-87f1-c72a96e42a9c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:04.563479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 415/825] 10.222.0.62 () {56 vars in 1124 bytes} [Fri Sep 23 13:44:04 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:04.568809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89a075f-9f70-45db-87f1-c72a96e42a9c None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:04.610448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 411/826] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:44:04 2022] PUT /identity/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/b471d768cd814ff8b4f6bd35d21e24a8/roles/809612fc6d62442cbcc179b5c4123465 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:44:04.610448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf13af48-7e1f-447c-98c1-68f4a19f7d3c None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:04.651593 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:04.703118 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89a075f-9f70-45db-87f1-c72a96e42a9c None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:04.734990 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf13af48-7e1f-447c-98c1-68f4a19f7d3c None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:04.747509 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0115de81-12c2-4760-b868-b1de8959f3f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:04.748242 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0115de81-12c2-4760-b868-b1de8959f3f2 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:04.748601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0115de81-12c2-4760-b868-b1de8959f3f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:04.748936 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0115de81-12c2-4760-b868-b1de8959f3f2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:04.761281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0115de81-12c2-4760-b868-b1de8959f3f2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:04.762975 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0115de81-12c2-4760-b868-b1de8959f3f2 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:04.779074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-0115de81-12c2-4760-b868-b1de8959f3f2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:04.779074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:04.779074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:04.779074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:04.779074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:04.779074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:04.779074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:04.779074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:04.779074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:04.779074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:04.779074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:04.779074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:04.779074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:04.779074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:04.779074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:04.779074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:04.780289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:04.780289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:04.780289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:04.780289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:04.780289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:04.780289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:04.780289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:04.780289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:04.780289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:04.780289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:04.780289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:04.780289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:04.780289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:04.780289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:04.780289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:04.780289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:04.780289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:04.781413 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:04.781413 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:04.781413 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:04.782372 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 416/827] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:04.782505 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ea4916-9e52-4096-aa3a-91b25e535109 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:04.783118 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ea4916-9e52-4096-aa3a-91b25e535109 None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:04.783496 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ea4916-9e52-4096-aa3a-91b25e535109 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:04.783866 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ea4916-9e52-4096-aa3a-91b25e535109 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:04.784633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ea4916-9e52-4096-aa3a-91b25e535109 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:04.786499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ea4916-9e52-4096-aa3a-91b25e535109 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:04.787463 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0115de81-12c2-4760-b868-b1de8959f3f2 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:04.866519 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 412/828] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 13:44:04 2022] POST /identity/v3/endpoints => generated 346 bytes in 266 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:44:04.911141 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0115de81-12c2-4760-b868-b1de8959f3f2 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:04.928203 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:04.961390 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa14bf24-6a61-40d6-9744-6cbb017378c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:04.962005 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa14bf24-6a61-40d6-9744-6cbb017378c6 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:04.962277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa14bf24-6a61-40d6-9744-6cbb017378c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:04.962572 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa14bf24-6a61-40d6-9744-6cbb017378c6 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:04.963222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa14bf24-6a61-40d6-9744-6cbb017378c6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:04.963573 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa14bf24-6a61-40d6-9744-6cbb017378c6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:04.964959 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa14bf24-6a61-40d6-9744-6cbb017378c6 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:04.979289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 417/829] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:04.984555 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ea4916-9e52-4096-aa3a-91b25e535109 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:05.042066 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:05.106590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ea4916-9e52-4096-aa3a-91b25e535109 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:05.142617 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd154ae0-fd2b-4ae2-9f1d-386f1690d2e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:05.143172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd154ae0-fd2b-4ae2-9f1d-386f1690d2e2 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:05.143523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd154ae0-fd2b-4ae2-9f1d-386f1690d2e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:05.143791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd154ae0-fd2b-4ae2-9f1d-386f1690d2e2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:05.151892 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd154ae0-fd2b-4ae2-9f1d-386f1690d2e2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:05.153444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd154ae0-fd2b-4ae2-9f1d-386f1690d2e2 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:05.165269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-bd154ae0-fd2b-4ae2-9f1d-386f1690d2e2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:05.165269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:05.165269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:05.165269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:05.165269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:05.165269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:05.165269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:05.165269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:05.165269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:05.165269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:05.165269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:05.165269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:05.165269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:44:05.165269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:44:05.165269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:44:05.166264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:44:05.166264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:05.166264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:05.166264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:05.166264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:05.166264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:05.166264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:05.166264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:05.166264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:05.166264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:05.166264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:05.166264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:05.166264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:05.166264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:05.166264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:05.166264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:05.166264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:05.167174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:44:05.167174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:44:05.167174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:44:05.167174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:44:05.167174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:44:05.167174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:44:05.167174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:05.167174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:05.167174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 413/830] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:44:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:05.172181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa14bf24-6a61-40d6-9744-6cbb017378c6 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:05.194500 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1e963f-a692-437a-b32b-5cfd24c8a327 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:05.194878 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1e963f-a692-437a-b32b-5cfd24c8a327 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:05.195153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1e963f-a692-437a-b32b-5cfd24c8a327 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:05.196753 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 414/831] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:05.203117 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb882cd-0647-435a-998b-47f9a62c15f9 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:05.203379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb882cd-0647-435a-998b-47f9a62c15f9 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:05.203711 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb882cd-0647-435a-998b-47f9a62c15f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:05.225812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-dcb882cd-0647-435a-998b-47f9a62c15f9 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:05.227500 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-dcb882cd-0647-435a-998b-47f9a62c15f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:05.280156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa14bf24-6a61-40d6-9744-6cbb017378c6 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:05.311564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 415/832] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:05 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 23 13:44:05.323735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbf3ffe-6382-40ff-8a4b-705b0249fc68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:05.324354 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbf3ffe-6382-40ff-8a4b-705b0249fc68 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:05.324632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbf3ffe-6382-40ff-8a4b-705b0249fc68 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:05.324899 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbf3ffe-6382-40ff-8a4b-705b0249fc68 None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:05.325594 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbf3ffe-6382-40ff-8a4b-705b0249fc68 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:05.325895 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbf3ffe-6382-40ff-8a4b-705b0249fc68 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:05.327397 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbf3ffe-6382-40ff-8a4b-705b0249fc68 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:05.340644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 418/833] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:44:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:05.346449 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb882cd-0647-435a-998b-47f9a62c15f9 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:05.387801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:05.459935 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb882cd-0647-435a-998b-47f9a62c15f9 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:05.495130 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40080c40-0238-4312-b1fe-0ba4ea41ccc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:05.495774 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40080c40-0238-4312-b1fe-0ba4ea41ccc5 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:05.496102 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40080c40-0238-4312-b1fe-0ba4ea41ccc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:05.496381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40080c40-0238-4312-b1fe-0ba4ea41ccc5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:05.497103 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40080c40-0238-4312-b1fe-0ba4ea41ccc5 None admin] RBAC: Adding query filter params (role.id=727f0fa598e54b9a929d2cf6277c875e, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=f01ed94dbb384a178ad0a45a6f3d8c17) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:05.497410 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40080c40-0238-4312-b1fe-0ba4ea41ccc5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:05.498858 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40080c40-0238-4312-b1fe-0ba4ea41ccc5 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:05.505699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 416/834] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:44:05 2022] GET /identity/v3/role_assignments?user.id=f01ed94dbb384a178ad0a45a6f3d8c17&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=727f0fa598e54b9a929d2cf6277c875e => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:05.606132 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbf3ffe-6382-40ff-8a4b-705b0249fc68 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:05.639107 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:05.721506 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbf3ffe-6382-40ff-8a4b-705b0249fc68 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:05.758983 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348c54ac-ffa2-4c80-9190-b3acbac43964 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:05.759614 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348c54ac-ffa2-4c80-9190-b3acbac43964 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:05.759894 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348c54ac-ffa2-4c80-9190-b3acbac43964 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:05.760166 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348c54ac-ffa2-4c80-9190-b3acbac43964 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:05.771564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-348c54ac-ffa2-4c80-9190-b3acbac43964 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:44:05.776803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-348c54ac-ffa2-4c80-9190-b3acbac43964 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:44:05.776803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:05.776803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:05.776803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:05.776803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:05.776803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:05.776803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:05.776803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:05.776803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:05.776803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:05.776803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:05.776803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:05.776803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:44:05.776803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:44:05.776803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:44:05.776803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:44:05.777699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:44:05.777699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:44:05.777699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:44:05.777699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:44:05.777699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:05.777699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:05.777699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:05.777699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:05.777699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:05.777699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:05.777699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:05.777699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:05.777699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:05.777699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:05.777699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:05.777699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:05.779125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:05.779125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:05.779125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:05.779125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:05.779125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:44:05.779125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:44:05.779125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:44:05.779125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:44:05.779125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:44:05.779125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:44:05.779125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:44:05.779125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:05.779125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 419/835] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:44:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:05.783812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40080c40-0238-4312-b1fe-0ba4ea41ccc5 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:05.893491 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40080c40-0238-4312-b1fe-0ba4ea41ccc5 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:05.895857 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:05.936079 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15dd237-d065-4701-8dfd-57b45b9ce476 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:05.936703 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15dd237-d065-4701-8dfd-57b45b9ce476 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:05.936985 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15dd237-d065-4701-8dfd-57b45b9ce476 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:05.937259 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15dd237-d065-4701-8dfd-57b45b9ce476 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:05.937936 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15dd237-d065-4701-8dfd-57b45b9ce476 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:05.938243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15dd237-d065-4701-8dfd-57b45b9ce476 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:05.939709 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15dd237-d065-4701-8dfd-57b45b9ce476 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:05.957681 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 417/836] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:44:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:05.963489 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348c54ac-ffa2-4c80-9190-b3acbac43964 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:06.059090 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:06.068942 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348c54ac-ffa2-4c80-9190-b3acbac43964 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:06.110295 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515f9d6d-7339-4250-aeb0-43d9cca7b082 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:06.111360 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515f9d6d-7339-4250-aeb0-43d9cca7b082 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:06.111834 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515f9d6d-7339-4250-aeb0-43d9cca7b082 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:06.112303 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515f9d6d-7339-4250-aeb0-43d9cca7b082 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:06.123717 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515f9d6d-7339-4250-aeb0-43d9cca7b082 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:06.125378 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-515f9d6d-7339-4250-aeb0-43d9cca7b082 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:06.140233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-515f9d6d-7339-4250-aeb0-43d9cca7b082 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:06.140233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:06.140233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:06.140233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:06.140233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:06.140233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:06.140233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:06.140233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:06.140233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:06.140233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:06.140233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:06.140233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:06.140233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:06.140233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:06.140233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:06.140233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:06.141149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:06.141149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:06.141149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:06.141149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:06.141149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:06.141149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:06.141149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:06.141149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:06.141149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:06.141149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:06.141149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:06.141149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:06.141149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:06.141149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:06.141149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:06.141149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:06.141149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:06.142021 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:06.142021 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:06.142021 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:06.144652 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 420/837] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:06.153186 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15dd237-d065-4701-8dfd-57b45b9ce476 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:06.255347 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15dd237-d065-4701-8dfd-57b45b9ce476 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:06.290376 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f51643-d3d8-4761-9be9-71ebb0c34d9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:06.291354 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f51643-d3d8-4761-9be9-71ebb0c34d9f None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:06.291828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f51643-d3d8-4761-9be9-71ebb0c34d9f None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:06.292311 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f51643-d3d8-4761-9be9-71ebb0c34d9f None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:06.293149 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f51643-d3d8-4761-9be9-71ebb0c34d9f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:06.293644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f51643-d3d8-4761-9be9-71ebb0c34d9f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:06.295209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f51643-d3d8-4761-9be9-71ebb0c34d9f None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:06.308421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 418/838] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:06.317638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515f9d6d-7339-4250-aeb0-43d9cca7b082 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:06.420920 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515f9d6d-7339-4250-aeb0-43d9cca7b082 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:06.457633 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb35c11-beba-4885-b418-5fa2962beb31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:06.458701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb35c11-beba-4885-b418-5fa2962beb31 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:06.459176 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb35c11-beba-4885-b418-5fa2962beb31 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:06.459661 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb35c11-beba-4885-b418-5fa2962beb31 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:06.468389 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb35c11-beba-4885-b418-5fa2962beb31 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:06.470037 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb35c11-beba-4885-b418-5fa2962beb31 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:06.481858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-6cb35c11-beba-4885-b418-5fa2962beb31 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 23 13:44:06.481858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:06.481858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:06.481858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:06.481858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:06.481858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:06.481858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:06.481858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:06.481858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:06.481858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:06.481858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:06.481858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:06.481858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:44:06.481858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:44:06.481858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:44:06.481858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:44:06.482828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:06.482828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:06.482828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:44:06.482828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:06.482828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:44:06.482828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:06.482828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:06.482828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:06.482828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:06.482828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:06.482828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:06.482828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:06.482828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:06.482828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:06.482828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:06.482828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:06.482828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:06.482828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:06.483806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:06.483806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:06.483806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:44:06.483806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:44:06.483806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:44:06.483806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:44:06.483806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:44:06.483806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:44:06.483806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 23 13:44:06.483806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:06.486408 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 421/839] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:44:06 2022] GET /identity/v3/users/swift => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:06.495421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f51643-d3d8-4761-9be9-71ebb0c34d9f None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:06.519935 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:06.611073 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f51643-d3d8-4761-9be9-71ebb0c34d9f None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:06.646925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2af477b-5f85-4e9b-bec8-e4648aaa5265 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:06.647589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2af477b-5f85-4e9b-bec8-e4648aaa5265 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:06.647868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2af477b-5f85-4e9b-bec8-e4648aaa5265 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:06.648176 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2af477b-5f85-4e9b-bec8-e4648aaa5265 None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:06.648955 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2af477b-5f85-4e9b-bec8-e4648aaa5265 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:06.649265 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2af477b-5f85-4e9b-bec8-e4648aaa5265 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:06.650704 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2af477b-5f85-4e9b-bec8-e4648aaa5265 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:06.669621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 419/840] 10.222.0.62 () {56 vars in 1135 bytes} [Fri Sep 23 13:44:06 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:06.674892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb35c11-beba-4885-b418-5fa2962beb31 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:06.686660 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:06.785347 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb35c11-beba-4885-b418-5fa2962beb31 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:06.824053 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c964e3bd-a84c-4a57-93e2-5642e0996cb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:06.824660 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c964e3bd-a84c-4a57-93e2-5642e0996cb0 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:06.824937 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c964e3bd-a84c-4a57-93e2-5642e0996cb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:06.825209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c964e3bd-a84c-4a57-93e2-5642e0996cb0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:06.896065 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c964e3bd-a84c-4a57-93e2-5642e0996cb0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:06.897438 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c964e3bd-a84c-4a57-93e2-5642e0996cb0 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:06.899428 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:06.904502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc227e04-f89e-420a-8e81-45a8aa65cda7 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:06.904781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc227e04-f89e-420a-8e81-45a8aa65cda7 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:06.905051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc227e04-f89e-420a-8e81-45a8aa65cda7 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:06.907189 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 420/841] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:44:06.912802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ca23ac-fc66-42f9-abd9-503b94e71230 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:06.913088 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ca23ac-fc66-42f9-abd9-503b94e71230 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:06.913358 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ca23ac-fc66-42f9-abd9-503b94e71230 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:06.932363 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-86ca23ac-fc66-42f9-abd9-503b94e71230 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:06.933774 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-86ca23ac-fc66-42f9-abd9-503b94e71230 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:07.010829 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 422/842] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:44:06 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 23 13:44:07.016576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c964e3bd-a84c-4a57-93e2-5642e0996cb0 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:07.027706 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 421/843] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:06 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 23 13:44:07.122413 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c964e3bd-a84c-4a57-93e2-5642e0996cb0 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:07.123941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:07.160371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77dece86-304c-4802-93fd-2512bcf775dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:07.160936 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dece86-304c-4802-93fd-2512bcf775dc None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:07.161215 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dece86-304c-4802-93fd-2512bcf775dc None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:07.161485 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77dece86-304c-4802-93fd-2512bcf775dc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:07.173641 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77dece86-304c-4802-93fd-2512bcf775dc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:07.175788 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77dece86-304c-4802-93fd-2512bcf775dc None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:07.193856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-77dece86-304c-4802-93fd-2512bcf775dc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:07.193856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:07.193856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:07.193856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:07.193856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:07.193856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:07.193856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:07.193856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:07.193856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:07.193856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:07.193856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:07.193856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:07.193856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:07.193856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:07.193856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:07.193856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:07.194864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:07.194864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:07.194864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:07.194864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:07.194864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:07.194864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:07.194864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:07.194864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:07.194864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:07.194864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:07.194864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:07.194864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:07.194864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:07.194864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:07.194864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:07.194864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:07.194864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:07.195737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:07.195737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:07.195737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:07.195737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 423/844] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:44:07.202939 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ca23ac-fc66-42f9-abd9-503b94e71230 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:07.222777 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:07.282147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4e6485-5429-4c25-814e-e163d1d1d06b None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:07.282494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4e6485-5429-4c25-814e-e163d1d1d06b None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:07.282726 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4e6485-5429-4c25-814e-e163d1d1d06b None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:07.286019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 424/845] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:07.290081 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124b8eec-f205-4048-b015-77088e5afbb8 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:07.290376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124b8eec-f205-4048-b015-77088e5afbb8 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:07.290648 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124b8eec-f205-4048-b015-77088e5afbb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:07.310287 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-124b8eec-f205-4048-b015-77088e5afbb8 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:07.311497 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-124b8eec-f205-4048-b015-77088e5afbb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:07.312735 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ca23ac-fc66-42f9-abd9-503b94e71230 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:07.352566 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403f9627-990f-4738-978b-a02aaf6943bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:07.353178 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403f9627-990f-4738-978b-a02aaf6943bd None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:07.353455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403f9627-990f-4738-978b-a02aaf6943bd None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:07.353725 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403f9627-990f-4738-978b-a02aaf6943bd None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:07.354779 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403f9627-990f-4738-978b-a02aaf6943bd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:07.355214 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403f9627-990f-4738-978b-a02aaf6943bd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:07.356930 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403f9627-990f-4738-978b-a02aaf6943bd None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:07.374691 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 422/846] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:07.374691 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403f9627-990f-4738-978b-a02aaf6943bd None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:07.388187 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 425/847] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:07 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 23 13:44:07.389038 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124b8eec-f205-4048-b015-77088e5afbb8 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:07.458726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:07.478374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:07.495513 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124b8eec-f205-4048-b015-77088e5afbb8 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:07.499406 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403f9627-990f-4738-978b-a02aaf6943bd None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:07.532026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7678afb4-f00c-4268-80e6-6abd3e668a6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:07.532952 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7678afb4-f00c-4268-80e6-6abd3e668a6a None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:07.533408 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7678afb4-f00c-4268-80e6-6abd3e668a6a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:07.533880 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7678afb4-f00c-4268-80e6-6abd3e668a6a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:07.544300 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7678afb4-f00c-4268-80e6-6abd3e668a6a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:07.546036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b9ee75-9c1f-4674-99fd-e2c742ac77a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:07.546317 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7678afb4-f00c-4268-80e6-6abd3e668a6a None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:07.546713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b9ee75-9c1f-4674-99fd-e2c742ac77a3 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:07.546989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b9ee75-9c1f-4674-99fd-e2c742ac77a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:07.547257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b9ee75-9c1f-4674-99fd-e2c742ac77a3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:07.560395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-7678afb4-f00c-4268-80e6-6abd3e668a6a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:07.560395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:07.560395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:07.560395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:07.560395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:07.560395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:07.560395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:07.560395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:07.560395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:07.560395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:07.560395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:07.560395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:07.560395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:44:07.560395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:44:07.560395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:44:07.561302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:44:07.561302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:07.561302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:07.561302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:07.561302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:07.561302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:07.561302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:07.561302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:07.561302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:07.561302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:07.561302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:07.561302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:07.561302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:07.561302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:07.561302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:07.561302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:07.561302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:07.562185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:44:07.562185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:44:07.562185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:44:07.562185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:44:07.562185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:44:07.562185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:44:07.562185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:07.562185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:07.563523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b9ee75-9c1f-4674-99fd-e2c742ac77a3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:44:07.564787 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 426/848] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:44:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:07.570895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-a3b9ee75-9c1f-4674-99fd-e2c742ac77a3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:44:07.570895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:07.570895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:07.570895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:07.570895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:07.570895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:07.570895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:07.570895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:07.570895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:07.570895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:07.570895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:07.570895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:07.570895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:44:07.570895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:44:07.570895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:44:07.570895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:44:07.571828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:44:07.571828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:44:07.571828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:44:07.571828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:44:07.571828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:07.571828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:07.571828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:07.571828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:07.571828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:07.571828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:07.571828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:07.571828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:07.571828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:07.571828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:07.571828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:07.571828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:07.572656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:07.572656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:07.572656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:07.572656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:07.572656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:44:07.572656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:44:07.572656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:44:07.572656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:44:07.572656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:44:07.572656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:44:07.572656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:44:07.572656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:07.574270 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7678afb4-f00c-4268-80e6-6abd3e668a6a None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:07.575232 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 423/849] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:44:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:07.583086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b9ee75-9c1f-4674-99fd-e2c742ac77a3 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:07.684761 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7678afb4-f00c-4268-80e6-6abd3e668a6a None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:07.708814 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b9ee75-9c1f-4674-99fd-e2c742ac77a3 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:07.719584 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260a86e1-ed7e-40e3-8ea1-f11969a5846c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:07.720212 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260a86e1-ed7e-40e3-8ea1-f11969a5846c None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:07.720453 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260a86e1-ed7e-40e3-8ea1-f11969a5846c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:07.720721 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260a86e1-ed7e-40e3-8ea1-f11969a5846c None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:07.721404 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260a86e1-ed7e-40e3-8ea1-f11969a5846c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:07.721706 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260a86e1-ed7e-40e3-8ea1-f11969a5846c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:07.725528 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260a86e1-ed7e-40e3-8ea1-f11969a5846c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:07.740869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:07.743607 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 427/850] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:44:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:07.744494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260a86e1-ed7e-40e3-8ea1-f11969a5846c None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:07.746433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14537dd-01b6-4b53-9985-131f5917e0f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:07.747052 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14537dd-01b6-4b53-9985-131f5917e0f1 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:07.747335 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14537dd-01b6-4b53-9985-131f5917e0f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:07.747609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14537dd-01b6-4b53-9985-131f5917e0f1 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:07.748278 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14537dd-01b6-4b53-9985-131f5917e0f1 None admin] RBAC: Adding query filter params (name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:07.748580 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14537dd-01b6-4b53-9985-131f5917e0f1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:07.749940 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14537dd-01b6-4b53-9985-131f5917e0f1 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:07.750061 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:07.759231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 424/851] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:44:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:07.767595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14537dd-01b6-4b53-9985-131f5917e0f1 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:07.867680 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14537dd-01b6-4b53-9985-131f5917e0f1 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:07.868358 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260a86e1-ed7e-40e3-8ea1-f11969a5846c None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:07.903306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de1bdca-2a59-4e4d-b1d6-352aeaeb88a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:07.903749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ef5746-66be-4031-b189-55899b1fa6ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:07.903936 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de1bdca-2a59-4e4d-b1d6-352aeaeb88a6 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:07.904181 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de1bdca-2a59-4e4d-b1d6-352aeaeb88a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:07.904366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ef5746-66be-4031-b189-55899b1fa6ae None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:07.904460 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de1bdca-2a59-4e4d-b1d6-352aeaeb88a6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:07.904646 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ef5746-66be-4031-b189-55899b1fa6ae None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:07.904916 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ef5746-66be-4031-b189-55899b1fa6ae None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:07.905151 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de1bdca-2a59-4e4d-b1d6-352aeaeb88a6 None admin] RBAC: Adding query filter params (role.id=727f0fa598e54b9a929d2cf6277c875e, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=4b9269509f1d454eab7a3008c2ba0039) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:07.905455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de1bdca-2a59-4e4d-b1d6-352aeaeb88a6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:07.905548 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ef5746-66be-4031-b189-55899b1fa6ae None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:07.907758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de1bdca-2a59-4e4d-b1d6-352aeaeb88a6 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:07.907860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ef5746-66be-4031-b189-55899b1fa6ae None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:07.919151 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 425/852] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:44:07 2022] GET /identity/v3/role_assignments?user.id=4b9269509f1d454eab7a3008c2ba0039&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=727f0fa598e54b9a929d2cf6277c875e => generated 608 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:07.919151 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de1bdca-2a59-4e4d-b1d6-352aeaeb88a6 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:07.919445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-27ef5746-66be-4031-b189-55899b1fa6ae None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 23 13:44:07.919445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:07.919445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:44:07.919445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:44:07.919445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:44:07.919445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:44:07.919445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:44:07.919445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:07.919445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 23 13:44:07.919445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:07.919445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:07.919445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:07.919445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:07.919445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:07.919445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:07.919445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:07.919445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:07.920772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:07.920772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:07.920772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:07.920772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:07.920772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:07.920772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:07.920772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:44:07.920772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:44:07.920772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:44:07.920772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:44:07.920772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:07.920772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:07.920772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:07.920772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:07.920772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:07.920772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:07.921638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:07.921638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:07.921638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:07.921638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:07.921638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:07.921638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:07.921638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:07.921638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:07.921638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:07.921638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:07.921638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:44:07.921638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:07.921638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 23 13:44:07.921638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:07.921638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 428/853] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:44:07 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:07.922452 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ac72eb-b9f3-457b-98db-8c41b6791ac1 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:07.922452 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ac72eb-b9f3-457b-98db-8c41b6791ac1 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:07.922452 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ac72eb-b9f3-457b-98db-8c41b6791ac1 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:07.925510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 429/854] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:07 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:07.925510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ac72eb-b9f3-457b-98db-8c41b6791ac1 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:07.996004 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:08.005273 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:08.041819 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de1bdca-2a59-4e4d-b1d6-352aeaeb88a6 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:08.045312 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ac72eb-b9f3-457b-98db-8c41b6791ac1 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:08.078963 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a75071-0edc-43cc-a3ae-8da8daee4970 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:08.079608 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a75071-0edc-43cc-a3ae-8da8daee4970 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:08.079889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a75071-0edc-43cc-a3ae-8da8daee4970 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:08.080162 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a75071-0edc-43cc-a3ae-8da8daee4970 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:08.086874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45d8d4f-a7c8-4b0f-b2b9-dc9887ccedf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:08.087772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45d8d4f-a7c8-4b0f-b2b9-dc9887ccedf4 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:08.088124 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45d8d4f-a7c8-4b0f-b2b9-dc9887ccedf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:08.088392 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45d8d4f-a7c8-4b0f-b2b9-dc9887ccedf4 None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:08.089033 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45d8d4f-a7c8-4b0f-b2b9-dc9887ccedf4 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:08.089387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45d8d4f-a7c8-4b0f-b2b9-dc9887ccedf4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:08.090783 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45d8d4f-a7c8-4b0f-b2b9-dc9887ccedf4 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:08.092355 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a75071-0edc-43cc-a3ae-8da8daee4970 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:08.093828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09a75071-0edc-43cc-a3ae-8da8daee4970 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:08.096266 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 430/855] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:44:07 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:08.096266 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d3c2e6-4276-4ebc-8600-7b83f56df235 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:08.096266 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d3c2e6-4276-4ebc-8600-7b83f56df235 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:08.096266 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d3c2e6-4276-4ebc-8600-7b83f56df235 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:08.112252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-09a75071-0edc-43cc-a3ae-8da8daee4970 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:08.112252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:08.112252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:08.112252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:08.112252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:08.112252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:08.112252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:08.112252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:08.112252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:08.112252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:08.112252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:08.112252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:08.112252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:08.112252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:08.112252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:08.112252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:08.113419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:08.113419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:08.113419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:08.113419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:08.113419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:08.113419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:08.113419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:08.113419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:08.113419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:08.113419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:08.113419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:08.113419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:08.113419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:08.113419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:08.113419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:08.113419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:08.113419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:08.114806 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:08.114806 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:08.114806 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:08.114806 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 426/856] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:08.115607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a75071-0edc-43cc-a3ae-8da8daee4970 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:08.123285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-b1d3c2e6-4276-4ebc-8600-7b83f56df235 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:08.124482 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-b1d3c2e6-4276-4ebc-8600-7b83f56df235 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:08.198366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 431/857] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:08 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:08.199352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d3c2e6-4276-4ebc-8600-7b83f56df235 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:08.235024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a75071-0edc-43cc-a3ae-8da8daee4970 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:08.246854 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:08.277401 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c49ab6b-a98f-4be0-8024-0e348d2bee1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:08.278595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c49ab6b-a98f-4be0-8024-0e348d2bee1c None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:08.279077 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c49ab6b-a98f-4be0-8024-0e348d2bee1c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:08.279573 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c49ab6b-a98f-4be0-8024-0e348d2bee1c None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:08.280963 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c49ab6b-a98f-4be0-8024-0e348d2bee1c None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:08.281738 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c49ab6b-a98f-4be0-8024-0e348d2bee1c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:08.284021 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c49ab6b-a98f-4be0-8024-0e348d2bee1c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:08.289110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:08.292179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 427/858] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:44:08 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:08.340847 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d3c2e6-4276-4ebc-8600-7b83f56df235 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:08.381896 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6ef1ac-243c-4c60-a04a-8a67d6655fd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:08.382953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6ef1ac-243c-4c60-a04a-8a67d6655fd5 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:08.383436 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6ef1ac-243c-4c60-a04a-8a67d6655fd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:08.383926 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6ef1ac-243c-4c60-a04a-8a67d6655fd5 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:08.384804 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6ef1ac-243c-4c60-a04a-8a67d6655fd5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:08.385311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6ef1ac-243c-4c60-a04a-8a67d6655fd5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:08.387088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6ef1ac-243c-4c60-a04a-8a67d6655fd5 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:08.401854 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 432/859] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:08.414381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c49ab6b-a98f-4be0-8024-0e348d2bee1c None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:08.494981 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6ef1ac-243c-4c60-a04a-8a67d6655fd5 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:08.534515 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c49ab6b-a98f-4be0-8024-0e348d2bee1c None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:08.580998 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a2a662-beb9-402c-ada3-9e0bb07202bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:08.581656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a2a662-beb9-402c-ada3-9e0bb07202bc None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:08.581935 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a2a662-beb9-402c-ada3-9e0bb07202bc None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:08.582194 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a2a662-beb9-402c-ada3-9e0bb07202bc None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:08.592955 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a2a662-beb9-402c-ada3-9e0bb07202bc None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:08.594585 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a2a662-beb9-402c-ada3-9e0bb07202bc None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:08.609822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-c3a2a662-beb9-402c-ada3-9e0bb07202bc None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 23 13:44:08.609822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:08.609822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:08.609822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:08.609822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:08.609822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:08.609822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:08.609822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:08.609822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:08.609822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:08.609822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:08.609822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:08.609822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:44:08.609822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:44:08.609822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:44:08.609822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:44:08.610875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:08.610875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:08.610875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:44:08.610875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:08.610875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:44:08.610875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:08.610875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:08.610875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:08.610875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:08.610875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:08.610875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:08.610875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:08.610875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:08.610875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:08.610875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:08.610875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:08.610875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:08.610875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:08.611805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:08.611805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:08.611805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:44:08.611805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:44:08.611805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:44:08.611805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:44:08.611805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:44:08.611805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:44:08.611805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 23 13:44:08.611805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:08.614570 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 428/860] 10.222.0.62 () {56 vars in 1117 bytes} [Fri Sep 23 13:44:08 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:44:08.614806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6ef1ac-243c-4c60-a04a-8a67d6655fd5 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:08.617492 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a2a662-beb9-402c-ada3-9e0bb07202bc None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:08.642497 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:08.658968 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c11725e-8c54-4920-9ead-1c8c9a5b923a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:08.660534 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c11725e-8c54-4920-9ead-1c8c9a5b923a None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:08.661245 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c11725e-8c54-4920-9ead-1c8c9a5b923a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:08.661714 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c11725e-8c54-4920-9ead-1c8c9a5b923a None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:08.662538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c11725e-8c54-4920-9ead-1c8c9a5b923a None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:08.663241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:08.666160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c11725e-8c54-4920-9ead-1c8c9a5b923a None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:08.679656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-4c11725e-8c54-4920-9ead-1c8c9a5b923a None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 23 13:44:08.679656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:08.679656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:44:08.679656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:44:08.679656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:44:08.679656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:44:08.679656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:44:08.679656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:08.679656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 23 13:44:08.679656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:08.679656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:08.679656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:08.679656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:08.679656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:08.679656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:08.679656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:08.680718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:08.680718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:08.680718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:08.680718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:08.680718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:08.680718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:08.680718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:08.680718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:44:08.680718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:44:08.680718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:44:08.680718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:44:08.680718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:08.680718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:08.680718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:08.680718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:08.680718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:08.680718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:08.681664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:08.681664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:08.681664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:08.681664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:08.681664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:08.681664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:08.681664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:08.681664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:08.681664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:08.681664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:08.681664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:44:08.681664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:08.681664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 23 13:44:08.681664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:08.690153 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 433/861] 10.222.0.62 () {56 vars in 1132 bytes} [Fri Sep 23 13:44:08 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:08.693070 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c11725e-8c54-4920-9ead-1c8c9a5b923a None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:08.754478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a2a662-beb9-402c-ada3-9e0bb07202bc None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:08.801402 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c222e052-415c-43c2-909b-bc8a73be3ad7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:08.802562 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c222e052-415c-43c2-909b-bc8a73be3ad7 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:08.803405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c222e052-415c-43c2-909b-bc8a73be3ad7 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:08.804166 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c222e052-415c-43c2-909b-bc8a73be3ad7 None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:08.805260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c222e052-415c-43c2-909b-bc8a73be3ad7 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:08.805791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c222e052-415c-43c2-909b-bc8a73be3ad7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:08.807982 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c222e052-415c-43c2-909b-bc8a73be3ad7 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:08.812302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c11725e-8c54-4920-9ead-1c8c9a5b923a None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:08.837490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 429/862] 10.222.0.62 () {56 vars in 1149 bytes} [Fri Sep 23 13:44:08 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:08.839463 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c222e052-415c-43c2-909b-bc8a73be3ad7 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:08.856256 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f403af09-3fc6-4a97-94ea-90d8b65fec5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:08.856891 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f403af09-3fc6-4a97-94ea-90d8b65fec5c None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:08.857166 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f403af09-3fc6-4a97-94ea-90d8b65fec5c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:08.857443 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f403af09-3fc6-4a97-94ea-90d8b65fec5c None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:08.858319 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f403af09-3fc6-4a97-94ea-90d8b65fec5c None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:08.858452 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f403af09-3fc6-4a97-94ea-90d8b65fec5c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:08.859841 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f403af09-3fc6-4a97-94ea-90d8b65fec5c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:08.871714 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 434/863] 10.222.0.62 () {56 vars in 1126 bytes} [Fri Sep 23 13:44:08 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:08.874876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f403af09-3fc6-4a97-94ea-90d8b65fec5c None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:08.932407 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:08.976038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:08.988903 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c222e052-415c-43c2-909b-bc8a73be3ad7 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:08.989063 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f403af09-3fc6-4a97-94ea-90d8b65fec5c None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:09.027008 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5430ab-6af6-46a2-b4d6-de6021e65a55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:09.027822 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5430ab-6af6-46a2-b4d6-de6021e65a55 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:09.028201 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5430ab-6af6-46a2-b4d6-de6021e65a55 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:09.028761 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5430ab-6af6-46a2-b4d6-de6021e65a55 None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:09.029323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5430ab-6af6-46a2-b4d6-de6021e65a55 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:09.029726 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5430ab-6af6-46a2-b4d6-de6021e65a55 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:09.031335 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5430ab-6af6-46a2-b4d6-de6021e65a55 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:09.035084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70933436-76a2-4235-b282-338ab4924786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:09.036272 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70933436-76a2-4235-b282-338ab4924786 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:09.036707 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70933436-76a2-4235-b282-338ab4924786 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:09.037146 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70933436-76a2-4235-b282-338ab4924786 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:09.046376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 435/864] 10.222.0.62 () {56 vars in 1126 bytes} [Fri Sep 23 13:44:08 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:09.129318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70933436-76a2-4235-b282-338ab4924786 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:09.130757 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70933436-76a2-4235-b282-338ab4924786 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:09.254727 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:09.266439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 430/865] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:44:08 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 23 13:44:09.269520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5430ab-6af6-46a2-b4d6-de6021e65a55 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:09.383358 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5430ab-6af6-46a2-b4d6-de6021e65a55 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:09.398235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:09.418751 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd0ebe3-2325-47e4-8688-012454f8f179 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:09.419591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd0ebe3-2325-47e4-8688-012454f8f179 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:09.420010 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd0ebe3-2325-47e4-8688-012454f8f179 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:09.420418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd0ebe3-2325-47e4-8688-012454f8f179 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:09.431421 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd0ebe3-2325-47e4-8688-012454f8f179 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:09.433179 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd0ebe3-2325-47e4-8688-012454f8f179 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:09.449361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-6fd0ebe3-2325-47e4-8688-012454f8f179 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:09.449361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:09.449361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:09.449361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:09.449361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:09.449361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:09.449361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:09.449361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:09.449361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:09.449361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:09.449361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:09.449361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:09.449361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:09.449361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:09.449361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:09.449361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:09.450493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:09.450493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:09.450493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:09.450493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:09.450493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:09.450493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:09.450493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:09.450493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:09.450493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:09.450493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:09.450493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:09.450493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:09.450493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:09.450493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:09.450493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:09.450493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:09.450493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:09.451586 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:09.451586 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:09.451586 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:09.457602 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 436/866] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:09.463414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70933436-76a2-4235-b282-338ab4924786 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:09.568121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70933436-76a2-4235-b282-338ab4924786 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:09.603657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791b798b-8fcc-4b4e-a895-ec6049751b82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:09.604321 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791b798b-8fcc-4b4e-a895-ec6049751b82 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:09.604603 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791b798b-8fcc-4b4e-a895-ec6049751b82 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:09.604936 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791b798b-8fcc-4b4e-a895-ec6049751b82 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:09.605517 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791b798b-8fcc-4b4e-a895-ec6049751b82 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:09.605819 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791b798b-8fcc-4b4e-a895-ec6049751b82 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:09.607207 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791b798b-8fcc-4b4e-a895-ec6049751b82 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:09.620479 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 431/867] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:09.626098 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd0ebe3-2325-47e4-8688-012454f8f179 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:09.733927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd0ebe3-2325-47e4-8688-012454f8f179 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:09.771793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608a2029-09e7-41a2-859c-febd0a8e0540 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:09.772823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608a2029-09e7-41a2-859c-febd0a8e0540 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:09.773329 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608a2029-09e7-41a2-859c-febd0a8e0540 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:09.773986 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608a2029-09e7-41a2-859c-febd0a8e0540 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:09.781482 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608a2029-09e7-41a2-859c-febd0a8e0540 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:09.783186 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608a2029-09e7-41a2-859c-febd0a8e0540 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:09.795394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-608a2029-09e7-41a2-859c-febd0a8e0540 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:09.795394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:09.795394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:09.795394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:09.795394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:09.795394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:09.795394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:09.795394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:09.795394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:09.795394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:09.795394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:09.795394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:09.795394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:44:09.795394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:44:09.795394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:44:09.796309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:44:09.796309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:09.796309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:09.796309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:09.796309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:09.796309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:09.796309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:09.796309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:09.796309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:09.796309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:09.796309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:09.796309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:09.796309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:09.796309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:09.796309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:09.796309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:09.796309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:09.797169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:44:09.797169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:44:09.797169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:44:09.797169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:44:09.797169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:44:09.797169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:44:09.797169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:09.797169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:09.803968 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 437/868] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:44:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:09.807397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791b798b-8fcc-4b4e-a895-ec6049751b82 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:09.863164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:09.914536 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791b798b-8fcc-4b4e-a895-ec6049751b82 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:09.950220 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c855fdc4-9408-4cf7-9c91-7ec55233429c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:09.951251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c855fdc4-9408-4cf7-9c91-7ec55233429c None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:09.951722 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c855fdc4-9408-4cf7-9c91-7ec55233429c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:09.952207 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c855fdc4-9408-4cf7-9c91-7ec55233429c None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:09.953104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c855fdc4-9408-4cf7-9c91-7ec55233429c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:09.953624 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c855fdc4-9408-4cf7-9c91-7ec55233429c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:09.955251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c855fdc4-9408-4cf7-9c91-7ec55233429c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:09.974279 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 432/869] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:44:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:09.979995 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608a2029-09e7-41a2-859c-febd0a8e0540 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:10.009941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:10.098721 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608a2029-09e7-41a2-859c-febd0a8e0540 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:10.136480 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2567a95-61de-4ba7-8731-d99416406409 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:10.137459 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2567a95-61de-4ba7-8731-d99416406409 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:10.137951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2567a95-61de-4ba7-8731-d99416406409 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:10.138441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2567a95-61de-4ba7-8731-d99416406409 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:10.139382 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2567a95-61de-4ba7-8731-d99416406409 None admin] RBAC: Adding query filter params (role.id=809612fc6d62442cbcc179b5c4123465, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=b471d768cd814ff8b4f6bd35d21e24a8) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:10.140068 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2567a95-61de-4ba7-8731-d99416406409 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:10.141667 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2567a95-61de-4ba7-8731-d99416406409 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:10.148646 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 438/870] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:44:09 2022] GET /identity/v3/role_assignments?user.id=b471d768cd814ff8b4f6bd35d21e24a8&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=809612fc6d62442cbcc179b5c4123465 => generated 608 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:10.156079 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c5ed95-0f9e-4bf9-a733-239d405a0abe None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:10.156721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c5ed95-0f9e-4bf9-a733-239d405a0abe None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:10.157190 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c5ed95-0f9e-4bf9-a733-239d405a0abe None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:10.158860 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 433/871] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:10.165290 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecefd14-d834-44ea-abb3-b7366025c441 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:10.165994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecefd14-d834-44ea-abb3-b7366025c441 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:10.166531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecefd14-d834-44ea-abb3-b7366025c441 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:10.187619 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-eecefd14-d834-44ea-abb3-b7366025c441 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:10.189239 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-eecefd14-d834-44ea-abb3-b7366025c441 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:10.274557 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 439/872] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:10 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:10.544281 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c5ed95-0f9e-4bf9-a733-239d405a0abe None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:10.551522 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353acd1e-9fd8-4ced-be63-c0c12676a30a None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:10.551796 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353acd1e-9fd8-4ced-be63-c0c12676a30a None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:10.552074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353acd1e-9fd8-4ced-be63-c0c12676a30a None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:10.553680 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 440/873] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:44:10.558691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93836910-06a0-4648-97a3-ffbfa844150f None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:10.558970 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93836910-06a0-4648-97a3-ffbfa844150f None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:10.559238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93836910-06a0-4648-97a3-ffbfa844150f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:10.577970 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-93836910-06a0-4648-97a3-ffbfa844150f None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:10.579190 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-93836910-06a0-4648-97a3-ffbfa844150f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:10.597594 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:10.663151 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 441/874] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:10 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 23 13:44:10.675511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c5ed95-0f9e-4bf9-a733-239d405a0abe None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:10.689844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:10.711593 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10627875-8eea-495f-8d10-d0d3c466e143 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:10.712221 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10627875-8eea-495f-8d10-d0d3c466e143 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:10.712499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10627875-8eea-495f-8d10-d0d3c466e143 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:10.712770 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10627875-8eea-495f-8d10-d0d3c466e143 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:10.713348 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10627875-8eea-495f-8d10-d0d3c466e143 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:10.714731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10627875-8eea-495f-8d10-d0d3c466e143 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:10.723480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-10627875-8eea-495f-8d10-d0d3c466e143 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 23 13:44:10.723480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:10.723480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:44:10.723480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:44:10.723480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:44:10.723480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:44:10.723480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:44:10.723480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:10.723480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 23 13:44:10.723480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:10.723480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:10.723480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:10.723480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:10.723480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:10.723480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:10.723480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:10.723480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:10.724467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:10.724467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:10.724467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:10.724467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:10.724467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:10.724467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:10.724467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:44:10.724467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:44:10.724467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:44:10.724467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:44:10.724467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:10.724467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:10.724467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:10.724467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:10.724467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:10.724467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:10.725307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:10.725307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:10.725307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:10.725307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:10.725307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:10.725307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:10.725307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:10.725307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:10.725307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:10.725307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:10.725307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:44:10.725307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:10.725307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 23 13:44:10.725307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:10.725307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 434/875] 10.222.0.62 () {56 vars in 1126 bytes} [Fri Sep 23 13:44:10 2022] GET /identity/v3/services/object-store => generated 93 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:10.731876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93836910-06a0-4648-97a3-ffbfa844150f None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:10.836596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93836910-06a0-4648-97a3-ffbfa844150f None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:10.871568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e05ff5f-9446-4853-95e7-17982c2a7766 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:10.872171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e05ff5f-9446-4853-95e7-17982c2a7766 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:10.872443 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e05ff5f-9446-4853-95e7-17982c2a7766 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:10.872764 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e05ff5f-9446-4853-95e7-17982c2a7766 None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:10.873392 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e05ff5f-9446-4853-95e7-17982c2a7766 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:10.873687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e05ff5f-9446-4853-95e7-17982c2a7766 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:10.875067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e05ff5f-9446-4853-95e7-17982c2a7766 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:10.884758 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 442/876] 10.222.0.62 () {56 vars in 1122 bytes} [Fri Sep 23 13:44:10 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:10.886206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10627875-8eea-495f-8d10-d0d3c466e143 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:10.957382 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e05ff5f-9446-4853-95e7-17982c2a7766 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:10.975427 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:10.997713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10627875-8eea-495f-8d10-d0d3c466e143 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:11.035663 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0e4d97-7000-4584-ad8a-e09e50fb2c79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:11.036478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0e4d97-7000-4584-ad8a-e09e50fb2c79 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:11.036766 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0e4d97-7000-4584-ad8a-e09e50fb2c79 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:11.037035 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0e4d97-7000-4584-ad8a-e09e50fb2c79 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:11.037799 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0e4d97-7000-4584-ad8a-e09e50fb2c79 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:11.038100 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0e4d97-7000-4584-ad8a-e09e50fb2c79 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:11.039499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d0e4d97-7000-4584-ad8a-e09e50fb2c79 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:11.044666 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 435/877] 10.222.0.62 () {56 vars in 1122 bytes} [Fri Sep 23 13:44:10 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:11.077786 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e05ff5f-9446-4853-95e7-17982c2a7766 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:11.117322 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56c5f15-888e-44d2-8707-e4c477963b59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:11.117865 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56c5f15-888e-44d2-8707-e4c477963b59 None admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:11.118222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56c5f15-888e-44d2-8707-e4c477963b59 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:11.118525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56c5f15-888e-44d2-8707-e4c477963b59 None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:11.119208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56c5f15-888e-44d2-8707-e4c477963b59 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:11.120622 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56c5f15-888e-44d2-8707-e4c477963b59 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:11.216770 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 443/878] 10.222.0.62 () {60 vars in 1141 bytes} [Fri Sep 23 13:44:10 2022] POST /identity/v3/services => generated 251 bytes in 260 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 13:44:11.418684 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2744f1-1709-4afd-b244-ff9b49b3cf79 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:11.419042 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2744f1-1709-4afd-b244-ff9b49b3cf79 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:11.419209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2744f1-1709-4afd-b244-ff9b49b3cf79 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:11.420478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 436/879] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:11.426751 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1751ccd-0f38-4961-a7e8-536ea2e63af0 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:11.427032 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1751ccd-0f38-4961-a7e8-536ea2e63af0 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:11.427304 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1751ccd-0f38-4961-a7e8-536ea2e63af0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:11.446398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-e1751ccd-0f38-4961-a7e8-536ea2e63af0 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:11.447653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-e1751ccd-0f38-4961-a7e8-536ea2e63af0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:11.522687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:11.529116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 444/880] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:11 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 23 13:44:11.805701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2744f1-1709-4afd-b244-ff9b49b3cf79 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:11.854051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:11.919679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2744f1-1709-4afd-b244-ff9b49b3cf79 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:11.959234 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0702448-41dd-4065-97c0-25570e3795f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:11.959864 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0702448-41dd-4065-97c0-25570e3795f9 None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:11.960145 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0702448-41dd-4065-97c0-25570e3795f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:11.960417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0702448-41dd-4065-97c0-25570e3795f9 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:11.961096 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0702448-41dd-4065-97c0-25570e3795f9 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:11.962522 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0702448-41dd-4065-97c0-25570e3795f9 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:12.064497 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 437/881] 10.222.0.62 () {60 vars in 1141 bytes} [Fri Sep 23 13:44:11 2022] POST /identity/v3/services => generated 269 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:44:12.499037 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636942fb-0947-4f54-a133-72f19d50d6d3 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:12.500011 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636942fb-0947-4f54-a133-72f19d50d6d3 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:12.500484 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636942fb-0947-4f54-a133-72f19d50d6d3 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:12.502308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 445/882] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:12.508589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e842c26f-bb0f-4eb4-98e2-4097ad6180ed None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:12.509103 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e842c26f-bb0f-4eb4-98e2-4097ad6180ed None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:12.509591 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e842c26f-bb0f-4eb4-98e2-4097ad6180ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:12.529490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-e842c26f-bb0f-4eb4-98e2-4097ad6180ed None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:12.531046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e842c26f-bb0f-4eb4-98e2-4097ad6180ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:12.605851 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 438/883] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:12 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 23 13:44:12.875782 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636942fb-0947-4f54-a133-72f19d50d6d3 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:12.983446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636942fb-0947-4f54-a133-72f19d50d6d3 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:13.026296 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea0fa73-257c-4a92-8ade-4680b0379a1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:13.027364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea0fa73-257c-4a92-8ade-4680b0379a1d None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:13.027853 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea0fa73-257c-4a92-8ade-4680b0379a1d None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:13.028328 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea0fa73-257c-4a92-8ade-4680b0379a1d None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:13.040368 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea0fa73-257c-4a92-8ade-4680b0379a1d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:44:13.045502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-9ea0fa73-257c-4a92-8ade-4680b0379a1d None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 23 13:44:13.045502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:13.045502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:13.045502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:13.045502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:13.045502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:13.045502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:13.045502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:13.045502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:13.045502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:13.045502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:13.045502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:13.045502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:44:13.045502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:44:13.045502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:44:13.045502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:44:13.046430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:44:13.046430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:44:13.046430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:44:13.046430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:44:13.046430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:13.046430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:13.046430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:13.046430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:13.046430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:13.046430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:13.046430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:13.046430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:13.046430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:13.046430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:13.046430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:13.046430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:13.047263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:13.047263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:13.047263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:13.047263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:13.047263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:44:13.047263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:44:13.047263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:44:13.047263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:44:13.047263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:44:13.047263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:44:13.047263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 23 13:44:13.047263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:13.050358 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 446/884] 10.222.0.62 () {56 vars in 1120 bytes} [Fri Sep 23 13:44:12 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:13.056502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e842c26f-bb0f-4eb4-98e2-4097ad6180ed None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:13.085586 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:13.161379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e842c26f-bb0f-4eb4-98e2-4097ad6180ed None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:13.200617 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db38feba-3b7e-450b-b09e-ea92e99dc33b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:13.201609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db38feba-3b7e-450b-b09e-ea92e99dc33b None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:13.202086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db38feba-3b7e-450b-b09e-ea92e99dc33b None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:13.202617 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db38feba-3b7e-450b-b09e-ea92e99dc33b None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:13.203481 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db38feba-3b7e-450b-b09e-ea92e99dc33b None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:13.203982 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db38feba-3b7e-450b-b09e-ea92e99dc33b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:13.205563 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db38feba-3b7e-450b-b09e-ea92e99dc33b None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:13.215420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 439/885] 10.222.0.62 () {56 vars in 1115 bytes} [Fri Sep 23 13:44:13 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:13.222413 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea0fa73-257c-4a92-8ade-4680b0379a1d None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:13.283255 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:13.302270 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caa7b61-9d10-44ff-8d01-04c545c08649 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:13.302949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caa7b61-9d10-44ff-8d01-04c545c08649 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:13.303283 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caa7b61-9d10-44ff-8d01-04c545c08649 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:13.304822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 440/886] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:13.310116 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d065c0d1-ff4d-4ab1-b2d6-5ce637a5183e None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:13.310487 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d065c0d1-ff4d-4ab1-b2d6-5ce637a5183e None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:13.310829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d065c0d1-ff4d-4ab1-b2d6-5ce637a5183e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:13.328235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea0fa73-257c-4a92-8ade-4680b0379a1d None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:13.335358 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-d065c0d1-ff4d-4ab1-b2d6-5ce637a5183e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:13.337140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d065c0d1-ff4d-4ab1-b2d6-5ce637a5183e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:13.371166 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843f24d6-ea58-47e3-b012-0cf4962449f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:13.372090 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843f24d6-ea58-47e3-b012-0cf4962449f1 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:13.372368 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843f24d6-ea58-47e3-b012-0cf4962449f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:13.372643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843f24d6-ea58-47e3-b012-0cf4962449f1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:13.389468 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843f24d6-ea58-47e3-b012-0cf4962449f1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:13.391867 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843f24d6-ea58-47e3-b012-0cf4962449f1 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:13.415561 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-843f24d6-ea58-47e3-b012-0cf4962449f1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:13.415561 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:13.415561 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:13.415561 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:13.415561 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:13.415561 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:13.415561 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:13.415561 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:13.415561 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:13.415561 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:13.415561 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:13.415561 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:13.415561 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:13.415561 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:13.415561 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:13.415561 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:13.416525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:13.416525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:13.416525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:13.416525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:13.416525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:13.416525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:13.416525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:13.416525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:13.416525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:13.416525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:13.416525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:13.416525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:13.416525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:13.416525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:13.416525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:13.416525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:13.416525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:13.417410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:13.417410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:13.417410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:13.417410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 447/887] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:13.423439 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843f24d6-ea58-47e3-b012-0cf4962449f1 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:13.438435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 441/888] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:13 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 23 13:44:13.495199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dab57f8-8304-4284-97cf-e2855b025b22 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:13.495844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dab57f8-8304-4284-97cf-e2855b025b22 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:13.496313 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dab57f8-8304-4284-97cf-e2855b025b22 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:13.498970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 442/889] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:13 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:44:13.505791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0428549-3bff-4f62-a3e8-fa9b2a95d874 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:13.506072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0428549-3bff-4f62-a3e8-fa9b2a95d874 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:13.506371 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0428549-3bff-4f62-a3e8-fa9b2a95d874 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:13.527673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-e0428549-3bff-4f62-a3e8-fa9b2a95d874 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:13.528923 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e0428549-3bff-4f62-a3e8-fa9b2a95d874 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:13.545827 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:13.550767 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843f24d6-ea58-47e3-b012-0cf4962449f1 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:13.564824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:13.588032 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327d1ca1-add8-4ca3-9dfa-977bdc0ff6f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:13.589082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327d1ca1-add8-4ca3-9dfa-977bdc0ff6f0 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:13.589555 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327d1ca1-add8-4ca3-9dfa-977bdc0ff6f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:13.589997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327d1ca1-add8-4ca3-9dfa-977bdc0ff6f0 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:13.590865 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327d1ca1-add8-4ca3-9dfa-977bdc0ff6f0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:13.592784 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327d1ca1-add8-4ca3-9dfa-977bdc0ff6f0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:13.593143 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-327d1ca1-add8-4ca3-9dfa-977bdc0ff6f0 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:13.606962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 448/890] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:13.612604 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327d1ca1-add8-4ca3-9dfa-977bdc0ff6f0 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:13.613758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 443/891] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:13 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 23 13:44:13.717425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-327d1ca1-add8-4ca3-9dfa-977bdc0ff6f0 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:13.722867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0428549-3bff-4f62-a3e8-fa9b2a95d874 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:13.762297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92c06a2-e87b-441e-ae80-11a2f05295df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:13.763305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92c06a2-e87b-441e-ae80-11a2f05295df None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:13.763805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92c06a2-e87b-441e-ae80-11a2f05295df None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:13.764291 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92c06a2-e87b-441e-ae80-11a2f05295df None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:13.774115 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92c06a2-e87b-441e-ae80-11a2f05295df None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:13.776287 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92c06a2-e87b-441e-ae80-11a2f05295df None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:13.791272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-d92c06a2-e87b-441e-ae80-11a2f05295df None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 23 13:44:13.791272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:13.791272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:13.791272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:13.791272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:13.791272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:13.791272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:13.791272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:13.791272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:13.791272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:13.791272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:13.791272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:13.791272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:44:13.791272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:44:13.791272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:44:13.791272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:44:13.792244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:13.792244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:13.792244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:44:13.792244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:13.792244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:44:13.792244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:13.792244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:13.792244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:13.792244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:13.792244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:13.792244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:13.792244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:13.792244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:13.792244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:13.792244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:13.792244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:13.792244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:13.792244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:13.793204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:13.793204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:13.793204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:44:13.793204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:44:13.793204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:44:13.793204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:44:13.793204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:44:13.793204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:44:13.793204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 23 13:44:13.793204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:13.797186 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 449/892] 10.222.0.62 () {56 vars in 1117 bytes} [Fri Sep 23 13:44:13 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:13.804006 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92c06a2-e87b-441e-ae80-11a2f05295df None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:13.840788 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:13.843457 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0428549-3bff-4f62-a3e8-fa9b2a95d874 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:13.887220 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98454fe4-decd-4404-86ae-128e2dce90fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:13.888296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98454fe4-decd-4404-86ae-128e2dce90fa None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:13.888631 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98454fe4-decd-4404-86ae-128e2dce90fa None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:13.889329 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98454fe4-decd-4404-86ae-128e2dce90fa None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:13.889998 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98454fe4-decd-4404-86ae-128e2dce90fa None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:13.891749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98454fe4-decd-4404-86ae-128e2dce90fa None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:13.914446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92c06a2-e87b-441e-ae80-11a2f05295df None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:13.947696 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcef2da-d448-42d9-a594-a97fbf384a3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:13.948298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcef2da-d448-42d9-a594-a97fbf384a3c None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:13.948575 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcef2da-d448-42d9-a594-a97fbf384a3c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:13.948844 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcef2da-d448-42d9-a594-a97fbf384a3c None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:13.953110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcef2da-d448-42d9-a594-a97fbf384a3c None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:13.953110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcef2da-d448-42d9-a594-a97fbf384a3c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:13.953110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcef2da-d448-42d9-a594-a97fbf384a3c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:13.970049 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 450/893] 10.222.0.62 () {56 vars in 1149 bytes} [Fri Sep 23 13:44:13 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:13.970645 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcef2da-d448-42d9-a594-a97fbf384a3c None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:13.986488 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 444/894] 10.222.0.62 () {60 vars in 1141 bytes} [Fri Sep 23 13:44:13 2022] POST /identity/v3/services => generated 241 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 13:44:13.986488 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98454fe4-decd-4404-86ae-128e2dce90fa None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:14.005404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:14.080754 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:14.091968 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcef2da-d448-42d9-a594-a97fbf384a3c None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:14.137593 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69530551-1133-4c38-9c12-54dedb95cb09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:14.138232 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69530551-1133-4c38-9c12-54dedb95cb09 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:14.138555 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69530551-1133-4c38-9c12-54dedb95cb09 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:14.138823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69530551-1133-4c38-9c12-54dedb95cb09 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:14.139523 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69530551-1133-4c38-9c12-54dedb95cb09 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:14.140960 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69530551-1133-4c38-9c12-54dedb95cb09 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:14.145796 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98454fe4-decd-4404-86ae-128e2dce90fa None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:14.151078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-69530551-1133-4c38-9c12-54dedb95cb09 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 23 13:44:14.151078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:14.151078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:44:14.151078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:44:14.151078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:44:14.151078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:44:14.151078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:44:14.151078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:14.151078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 23 13:44:14.151078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:14.151078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:14.151078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:14.151078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:14.151078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:14.151078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:14.151078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:14.151078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:14.152387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:14.152387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:14.152387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:14.152387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:14.152387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:14.152387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:14.152387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:44:14.152387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:44:14.152387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:44:14.152387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:44:14.152387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:14.152387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:14.152387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:14.152387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:14.152387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:14.152387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:14.153293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:14.153293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:14.153293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:14.153293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:14.153293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:14.153293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:14.153293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:14.153293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:14.153293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:14.153293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:14.153293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:44:14.153293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:14.153293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 23 13:44:14.153293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:14.153293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 451/895] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:44:13 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:14.158666 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69530551-1133-4c38-9c12-54dedb95cb09 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:14.195879 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def950e7-bdca-4131-b31d-98b79c81adbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:14.197887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def950e7-bdca-4131-b31d-98b79c81adbd None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:14.198934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def950e7-bdca-4131-b31d-98b79c81adbd None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:14.199597 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def950e7-bdca-4131-b31d-98b79c81adbd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:14.285064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69530551-1133-4c38-9c12-54dedb95cb09 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:14.297425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def950e7-bdca-4131-b31d-98b79c81adbd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:14.299101 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def950e7-bdca-4131-b31d-98b79c81adbd None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:14.311992 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:14.337732 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfed301-cf0a-46cb-bf7c-84e3a159d309 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:14.339034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfed301-cf0a-46cb-bf7c-84e3a159d309 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:14.339296 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfed301-cf0a-46cb-bf7c-84e3a159d309 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:14.339559 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dfed301-cf0a-46cb-bf7c-84e3a159d309 None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:14.340197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dfed301-cf0a-46cb-bf7c-84e3a159d309 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:14.340524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dfed301-cf0a-46cb-bf7c-84e3a159d309 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:14.341894 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dfed301-cf0a-46cb-bf7c-84e3a159d309 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:14.350366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 452/896] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:44:14 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:14.353534 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfed301-cf0a-46cb-bf7c-84e3a159d309 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:14.375199 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:14.416410 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 445/897] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:44:13 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 23 13:44:14.417925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bd354e-de25-4fbe-a275-cf74d1723bc8 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:14.418214 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bd354e-de25-4fbe-a275-cf74d1723bc8 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:14.418515 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bd354e-de25-4fbe-a275-cf74d1723bc8 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:14.419919 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 446/898] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:14.422448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bd354e-de25-4fbe-a275-cf74d1723bc8 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:14.472701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dfed301-cf0a-46cb-bf7c-84e3a159d309 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:14.509480 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073849b9-5662-4ec6-9f71-7c5fdb73e017 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:14.510083 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073849b9-5662-4ec6-9f71-7c5fdb73e017 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:14.510375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073849b9-5662-4ec6-9f71-7c5fdb73e017 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:14.510644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073849b9-5662-4ec6-9f71-7c5fdb73e017 None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:14.511263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073849b9-5662-4ec6-9f71-7c5fdb73e017 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:14.511559 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073849b9-5662-4ec6-9f71-7c5fdb73e017 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:14.512909 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073849b9-5662-4ec6-9f71-7c5fdb73e017 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:14.517752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 453/899] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:44:14 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:14.519037 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144fbf5a-0e9a-41f1-bce0-f3528698aa1b None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:14.519318 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144fbf5a-0e9a-41f1-bce0-f3528698aa1b None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:14.519584 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144fbf5a-0e9a-41f1-bce0-f3528698aa1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:14.536882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2bd354e-de25-4fbe-a275-cf74d1723bc8 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:14.537179 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-144fbf5a-0e9a-41f1-bce0-f3528698aa1b None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:14.538376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-144fbf5a-0e9a-41f1-bce0-f3528698aa1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:14.547712 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:14.575407 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4639c11c-5040-4e78-b15c-fa655cd0818c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:14.576157 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4639c11c-5040-4e78-b15c-fa655cd0818c None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:14.576425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4639c11c-5040-4e78-b15c-fa655cd0818c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:14.576695 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4639c11c-5040-4e78-b15c-fa655cd0818c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:14.595907 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4639c11c-5040-4e78-b15c-fa655cd0818c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:14.597981 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4639c11c-5040-4e78-b15c-fa655cd0818c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:14.618365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 454/900] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:14 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:14.618365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144fbf5a-0e9a-41f1-bce0-f3528698aa1b None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:14.618978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-4639c11c-5040-4e78-b15c-fa655cd0818c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:14.618978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:14.618978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:14.618978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:14.618978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:14.618978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:14.618978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:14.618978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:14.618978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:14.618978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:14.618978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:14.618978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:14.618978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:14.618978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:14.618978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:14.618978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:14.620054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:14.620054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:14.620054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:14.620054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:14.620054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:14.620054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:14.620054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:14.620054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:14.620054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:14.620054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:14.620054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:14.620054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:14.620054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:14.620054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:14.620054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:14.620054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:14.620054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:14.621144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:14.621144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:14.621144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:14.622737 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 447/901] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:44:14.623980 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4639c11c-5040-4e78-b15c-fa655cd0818c None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:14.624373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:14.738055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144fbf5a-0e9a-41f1-bce0-f3528698aa1b None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:14.747130 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4639c11c-5040-4e78-b15c-fa655cd0818c None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:14.774998 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5822fd0-5b63-4f4c-8960-bbd40485ec58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:14.775608 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5822fd0-5b63-4f4c-8960-bbd40485ec58 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:14.775883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5822fd0-5b63-4f4c-8960-bbd40485ec58 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:14.776149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5822fd0-5b63-4f4c-8960-bbd40485ec58 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:14.776916 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5822fd0-5b63-4f4c-8960-bbd40485ec58 None admin] RBAC: Adding query filter params (interface=public, service_id=1e39f31e9322434eac34199a7734afe0, region_id=RegionOne) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:14.777218 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5822fd0-5b63-4f4c-8960-bbd40485ec58 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:14.778597 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5822fd0-5b63-4f4c-8960-bbd40485ec58 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:14.784816 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 455/902] 10.222.0.62 () {56 vars in 1251 bytes} [Fri Sep 23 13:44:14 2022] GET /identity/v3/endpoints?service_id=1e39f31e9322434eac34199a7734afe0&interface=public®ion_id=RegionOne => generated 194 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:14.789816 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5822fd0-5b63-4f4c-8960-bbd40485ec58 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:14.794130 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52baef32-6514-4768-a700-f00b41c26bf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:14.794779 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52baef32-6514-4768-a700-f00b41c26bf0 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:14.795056 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52baef32-6514-4768-a700-f00b41c26bf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:14.795322 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52baef32-6514-4768-a700-f00b41c26bf0 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:14.795962 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52baef32-6514-4768-a700-f00b41c26bf0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:14.796261 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52baef32-6514-4768-a700-f00b41c26bf0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:14.797670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52baef32-6514-4768-a700-f00b41c26bf0 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:14.817369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 448/903] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:14.822248 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52baef32-6514-4768-a700-f00b41c26bf0 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:14.851956 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:14.878883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:14.910805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5822fd0-5b63-4f4c-8960-bbd40485ec58 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:14.934053 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52baef32-6514-4768-a700-f00b41c26bf0 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:14.946742 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbf5dc7-90ec-4ca2-900c-9efc5154c828 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:14.947342 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbf5dc7-90ec-4ca2-900c-9efc5154c828 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:14.947625 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbf5dc7-90ec-4ca2-900c-9efc5154c828 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:14.947883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecbf5dc7-90ec-4ca2-900c-9efc5154c828 None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:14.948477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbf5dc7-90ec-4ca2-900c-9efc5154c828 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:14.949829 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecbf5dc7-90ec-4ca2-900c-9efc5154c828 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:14.958452 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 456/904] 10.222.0.62 () {56 vars in 1087 bytes} [Fri Sep 23 13:44:14 2022] GET /identity/v3/services => generated 1497 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 13:44:14.958452 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbf5dc7-90ec-4ca2-900c-9efc5154c828 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:14.979350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8d7c15-3327-4efd-91fc-b5c28c4d15cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:14.980437 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8d7c15-3327-4efd-91fc-b5c28c4d15cc None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:14.980868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8d7c15-3327-4efd-91fc-b5c28c4d15cc None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:14.981403 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8d7c15-3327-4efd-91fc-b5c28c4d15cc None admin] PATH_INFO: `/v3/projects/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:14.990737 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8d7c15-3327-4efd-91fc-b5c28c4d15cc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:14.992506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8d7c15-3327-4efd-91fc-b5c28c4d15cc None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:15.006474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-1b8d7c15-3327-4efd-91fc-b5c28c4d15cc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:15.006474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:15.006474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:15.006474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:15.006474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:15.006474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:15.006474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:15.006474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:15.006474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:15.006474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:15.006474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:15.006474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:15.006474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:44:15.006474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:44:15.006474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:44:15.007473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:44:15.007473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:15.007473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:15.007473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:15.007473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:15.007473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:15.007473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:15.007473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:15.007473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:15.007473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:15.007473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:15.007473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:15.007473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:15.007473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:15.007473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:15.007473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:15.007473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:15.008414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:44:15.008414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:44:15.008414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:44:15.008414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:44:15.008414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:44:15.008414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:44:15.008414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:15.008414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:15.008414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 449/905] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:44:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:44:15.013979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8d7c15-3327-4efd-91fc-b5c28c4d15cc None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:15.093877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecbf5dc7-90ec-4ca2-900c-9efc5154c828 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:15.108963 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:15.132335 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8d7c15-3327-4efd-91fc-b5c28c4d15cc None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:15.146036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcdfa48-9354-41e3-97f2-da160654b5f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:15.147286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcdfa48-9354-41e3-97f2-da160654b5f7 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:15.155787 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcdfa48-9354-41e3-97f2-da160654b5f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:15.155787 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcdfa48-9354-41e3-97f2-da160654b5f7 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:15.155787 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcdfa48-9354-41e3-97f2-da160654b5f7 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:15.155787 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcdfa48-9354-41e3-97f2-da160654b5f7 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:15.155787 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:15.164778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-3bcdfa48-9354-41e3-97f2-da160654b5f7 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 23 13:44:15.164778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:15.164778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:44:15.164778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:44:15.164778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:44:15.164778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:44:15.164778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:44:15.164778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:15.164778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 23 13:44:15.164778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:15.164778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:15.164778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:15.164778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:15.164778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:15.164778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:15.164778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:15.165756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:15.165756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:15.165756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:15.165756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:15.165756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:15.165756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:15.165756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:15.165756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:44:15.165756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:44:15.165756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:44:15.165756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:44:15.165756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:15.165756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:15.165756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:15.165756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:15.165756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:15.165756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:15.166687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:15.166687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:15.166687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:15.166687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:15.166687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:15.166687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:15.166687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:15.166687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:15.166687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:15.166687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:15.166687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:44:15.166687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:15.166687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 23 13:44:15.166687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:15.166687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 457/906] 10.222.0.62 () {56 vars in 1132 bytes} [Fri Sep 23 13:44:14 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:15.171729 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcdfa48-9354-41e3-97f2-da160654b5f7 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:15.175780 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857e411e-4853-46c9-8faa-b9c27849397d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:15.176390 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857e411e-4853-46c9-8faa-b9c27849397d None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:15.176667 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857e411e-4853-46c9-8faa-b9c27849397d None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:15.176931 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857e411e-4853-46c9-8faa-b9c27849397d None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:15.177616 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857e411e-4853-46c9-8faa-b9c27849397d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:15.177915 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857e411e-4853-46c9-8faa-b9c27849397d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:15.179965 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-857e411e-4853-46c9-8faa-b9c27849397d None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:15.193842 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 450/907] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:44:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:15.203162 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857e411e-4853-46c9-8faa-b9c27849397d None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:15.299224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcdfa48-9354-41e3-97f2-da160654b5f7 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:15.318773 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857e411e-4853-46c9-8faa-b9c27849397d None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:15.347947 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b27e305-4a72-4ae8-bf9b-275052bbb33c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:15.348480 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b27e305-4a72-4ae8-bf9b-275052bbb33c None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:15.348755 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b27e305-4a72-4ae8-bf9b-275052bbb33c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:15.349021 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b27e305-4a72-4ae8-bf9b-275052bbb33c None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:15.349650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b27e305-4a72-4ae8-bf9b-275052bbb33c None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:15.349946 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b27e305-4a72-4ae8-bf9b-275052bbb33c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:15.352071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b27e305-4a72-4ae8-bf9b-275052bbb33c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:15.360596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 458/908] 10.222.0.62 () {56 vars in 1126 bytes} [Fri Sep 23 13:44:15 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:15.370453 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b27e305-4a72-4ae8-bf9b-275052bbb33c None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:15.371325 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1caba4-6f95-405d-9d57-098b7e153d65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:15.372138 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1caba4-6f95-405d-9d57-098b7e153d65 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:15.372609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1caba4-6f95-405d-9d57-098b7e153d65 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:15.373065 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1caba4-6f95-405d-9d57-098b7e153d65 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:15.373960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1caba4-6f95-405d-9d57-098b7e153d65 None admin] RBAC: Adding query filter params (role.id=b2084c06d5a44297a9a7795d559fae94, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=b471d768cd814ff8b4f6bd35d21e24a8) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:15.374488 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1caba4-6f95-405d-9d57-098b7e153d65 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:15.376077 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1caba4-6f95-405d-9d57-098b7e153d65 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:15.383436 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 451/909] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:44:15 2022] GET /identity/v3/role_assignments?user.id=b471d768cd814ff8b4f6bd35d21e24a8&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=b2084c06d5a44297a9a7795d559fae94 => generated 259 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:15.447968 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:15.491650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b27e305-4a72-4ae8-bf9b-275052bbb33c None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:15.534279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e824ea63-3405-45d8-8f87-4a9cac201948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:15.534927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e824ea63-3405-45d8-8f87-4a9cac201948 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:15.535201 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e824ea63-3405-45d8-8f87-4a9cac201948 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:15.535475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e824ea63-3405-45d8-8f87-4a9cac201948 None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:15.536118 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e824ea63-3405-45d8-8f87-4a9cac201948 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:15.536417 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e824ea63-3405-45d8-8f87-4a9cac201948 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:15.537799 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e824ea63-3405-45d8-8f87-4a9cac201948 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:15.543807 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 459/910] 10.222.0.62 () {56 vars in 1126 bytes} [Fri Sep 23 13:44:15 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:15.549989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1caba4-6f95-405d-9d57-098b7e153d65 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:15.553994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:15.661494 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1caba4-6f95-405d-9d57-098b7e153d65 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:15.696770 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f46264-32fe-4b1d-b09b-0c5eb149a2c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:15.697637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f46264-32fe-4b1d-b09b-0c5eb149a2c4 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:15.698064 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f46264-32fe-4b1d-b09b-0c5eb149a2c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:15.698502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f46264-32fe-4b1d-b09b-0c5eb149a2c4 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:15.699332 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f46264-32fe-4b1d-b09b-0c5eb149a2c4 None admin] RBAC: Adding query filter params (interface=public, service_id=a54dfbe5b62e4c6d9b0def972b7fefba, region_id=RegionOne) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:15.699782 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f46264-32fe-4b1d-b09b-0c5eb149a2c4 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:15.701292 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f46264-32fe-4b1d-b09b-0c5eb149a2c4 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:15.707013 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 452/911] 10.222.0.62 () {56 vars in 1251 bytes} [Fri Sep 23 13:44:15 2022] GET /identity/v3/endpoints?service_id=a54dfbe5b62e4c6d9b0def972b7fefba&interface=public®ion_id=RegionOne => generated 194 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:15.718380 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e824ea63-3405-45d8-8f87-4a9cac201948 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:15.824799 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e824ea63-3405-45d8-8f87-4a9cac201948 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:15.853611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:15.863128 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2cda12d-5615-4192-a446-80f7e66d0bf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:15.864074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2cda12d-5615-4192-a446-80f7e66d0bf7 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:15.864592 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2cda12d-5615-4192-a446-80f7e66d0bf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:15.865055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2cda12d-5615-4192-a446-80f7e66d0bf7 None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:15.865831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2cda12d-5615-4192-a446-80f7e66d0bf7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:15.867680 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2cda12d-5615-4192-a446-80f7e66d0bf7 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:15.874211 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 460/912] 10.222.0.62 () {56 vars in 1087 bytes} [Fri Sep 23 13:44:15 2022] GET /identity/v3/services => generated 1497 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 13:44:16.357850 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8350a6e9-3da1-4dfd-9ef2-b12190b12d70 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:16.358439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8350a6e9-3da1-4dfd-9ef2-b12190b12d70 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:16.358439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8350a6e9-3da1-4dfd-9ef2-b12190b12d70 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:16.360121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 453/913] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:16.367063 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a036e5d4-27b9-4c20-9e47-c09571d39857 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:16.367346 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a036e5d4-27b9-4c20-9e47-c09571d39857 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:16.367620 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a036e5d4-27b9-4c20-9e47-c09571d39857 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:16.385899 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-a036e5d4-27b9-4c20-9e47-c09571d39857 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:16.387191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a036e5d4-27b9-4c20-9e47-c09571d39857 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:16.456497 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 461/914] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:16 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:16.732473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8350a6e9-3da1-4dfd-9ef2-b12190b12d70 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:16.804283 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:16.842446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8350a6e9-3da1-4dfd-9ef2-b12190b12d70 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:16.878869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8196c64-44bc-43ae-a329-767bb5e177d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:16.879502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8196c64-44bc-43ae-a329-767bb5e177d4 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:16.879781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8196c64-44bc-43ae-a329-767bb5e177d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:16.880053 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8196c64-44bc-43ae-a329-767bb5e177d4 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:16.880633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8196c64-44bc-43ae-a329-767bb5e177d4 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:16.881993 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8196c64-44bc-43ae-a329-767bb5e177d4 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:16.895255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-c8196c64-44bc-43ae-a329-767bb5e177d4 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 23 13:44:16.895255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:16.895255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:44:16.895255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:44:16.895255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:44:16.895255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:44:16.895255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:44:16.895255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:16.895255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 23 13:44:16.895255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:16.895255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:16.895255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:16.895255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:16.895255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:16.895255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:16.895255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:16.895255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:16.896216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:16.896216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:16.896216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:16.896216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:16.896216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:16.896216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:16.896216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:44:16.896216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:44:16.896216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:44:16.896216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:44:16.896216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:16.896216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:16.896216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:16.896216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:16.896216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:16.896216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:16.897048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:16.897048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:16.897048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:16.897048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:16.897048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:16.897048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:16.897048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:16.897048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:16.897048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:16.897048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:16.897048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:44:16.897048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:16.897048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 23 13:44:16.897048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:16.899462 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 454/915] 10.222.0.62 () {56 vars in 1126 bytes} [Fri Sep 23 13:44:16 2022] GET /identity/v3/services/object-store => generated 93 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:16.905666 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a036e5d4-27b9-4c20-9e47-c09571d39857 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:17.009807 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a036e5d4-27b9-4c20-9e47-c09571d39857 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:17.032853 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:17.045003 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fd897e-510e-4cec-92ee-863b354aebc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:17.049927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fd897e-510e-4cec-92ee-863b354aebc9 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:17.049927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fd897e-510e-4cec-92ee-863b354aebc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:17.049927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fd897e-510e-4cec-92ee-863b354aebc9 None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:17.049927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fd897e-510e-4cec-92ee-863b354aebc9 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:17.049927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fd897e-510e-4cec-92ee-863b354aebc9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:17.049927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fd897e-510e-4cec-92ee-863b354aebc9 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:17.054664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 462/916] 10.222.0.62 () {56 vars in 1122 bytes} [Fri Sep 23 13:44:16 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:17.060045 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8196c64-44bc-43ae-a329-767bb5e177d4 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:17.163600 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8196c64-44bc-43ae-a329-767bb5e177d4 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:17.194306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:17.199291 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e06b420-41da-4324-af3c-d78a446ceec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:17.199821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e06b420-41da-4324-af3c-d78a446ceec3 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:17.200153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e06b420-41da-4324-af3c-d78a446ceec3 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:17.200429 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e06b420-41da-4324-af3c-d78a446ceec3 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:17.201062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e06b420-41da-4324-af3c-d78a446ceec3 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:17.201645 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e06b420-41da-4324-af3c-d78a446ceec3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:17.203032 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e06b420-41da-4324-af3c-d78a446ceec3 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:17.208543 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 455/917] 10.222.0.62 () {56 vars in 1122 bytes} [Fri Sep 23 13:44:17 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:17.217861 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fd897e-510e-4cec-92ee-863b354aebc9 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:17.334694 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fd897e-510e-4cec-92ee-863b354aebc9 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:17.358046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b8a6e-41e8-4245-aad3-ec8868cb8ea8 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:17.358605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b8a6e-41e8-4245-aad3-ec8868cb8ea8 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:17.358605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b8a6e-41e8-4245-aad3-ec8868cb8ea8 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:17.360206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 456/918] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:17.365712 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3f1d59-4e9d-44ff-9223-1baf01f95b87 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:17.365979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3f1d59-4e9d-44ff-9223-1baf01f95b87 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:17.366561 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3f1d59-4e9d-44ff-9223-1baf01f95b87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:17.373703 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27ca8ec-afbf-4c3e-aca9-bea33aee3be8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:17.374682 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27ca8ec-afbf-4c3e-aca9-bea33aee3be8 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:17.375188 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27ca8ec-afbf-4c3e-aca9-bea33aee3be8 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:17.375671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27ca8ec-afbf-4c3e-aca9-bea33aee3be8 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:17.376653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27ca8ec-afbf-4c3e-aca9-bea33aee3be8 None admin] RBAC: Adding query filter params (interface=public, service_id=f6cccd6a7e1b4c8499a00c81c7d2a9ef, region_id=RegionOne) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:17.377192 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27ca8ec-afbf-4c3e-aca9-bea33aee3be8 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:17.379509 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27ca8ec-afbf-4c3e-aca9-bea33aee3be8 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:17.385572 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 463/919] 10.222.0.62 () {56 vars in 1251 bytes} [Fri Sep 23 13:44:17 2022] GET /identity/v3/endpoints?service_id=f6cccd6a7e1b4c8499a00c81c7d2a9ef&interface=public®ion_id=RegionOne => generated 194 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:17.385836 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-7e3f1d59-4e9d-44ff-9223-1baf01f95b87 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:17.387033 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-7e3f1d59-4e9d-44ff-9223-1baf01f95b87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:17.390723 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27ca8ec-afbf-4c3e-aca9-bea33aee3be8 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:17.446292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:17.462610 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 457/920] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:17 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 23 13:44:17.499721 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27ca8ec-afbf-4c3e-aca9-bea33aee3be8 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:17.535207 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a08c876-f320-48ac-bbcb-9b09514fb87e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:17.536267 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a08c876-f320-48ac-bbcb-9b09514fb87e None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:17.536754 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a08c876-f320-48ac-bbcb-9b09514fb87e None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:17.537236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a08c876-f320-48ac-bbcb-9b09514fb87e None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:17.538091 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a08c876-f320-48ac-bbcb-9b09514fb87e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:17.539670 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a08c876-f320-48ac-bbcb-9b09514fb87e None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:17.546219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 464/921] 10.222.0.62 () {56 vars in 1087 bytes} [Fri Sep 23 13:44:17 2022] GET /identity/v3/services => generated 1497 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 13:44:17.709450 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed6f7a1-f488-4ac8-949a-f2ed9101bd2a None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:17.709806 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed6f7a1-f488-4ac8-949a-f2ed9101bd2a None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:17.709995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed6f7a1-f488-4ac8-949a-f2ed9101bd2a None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:17.711689 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 458/922] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:17.717166 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bbb427-0769-458e-9cf8-f6880dc5c5cd None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:17.717462 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bbb427-0769-458e-9cf8-f6880dc5c5cd None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:17.717725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bbb427-0769-458e-9cf8-f6880dc5c5cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:17.728938 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed6f7a1-f488-4ac8-949a-f2ed9101bd2a None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:17.739565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c1bbb427-0769-458e-9cf8-f6880dc5c5cd None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:17.740814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c1bbb427-0769-458e-9cf8-f6880dc5c5cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:17.816317 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 465/923] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:17 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:17.844656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed6f7a1-f488-4ac8-949a-f2ed9101bd2a None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:17.871581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:17.881247 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfafed0e-b206-4976-94ae-cdb0d7da1d23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:17.890247 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfafed0e-b206-4976-94ae-cdb0d7da1d23 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:17.890247 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfafed0e-b206-4976-94ae-cdb0d7da1d23 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:17.890247 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfafed0e-b206-4976-94ae-cdb0d7da1d23 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:17.890247 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfafed0e-b206-4976-94ae-cdb0d7da1d23 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:17.890247 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfafed0e-b206-4976-94ae-cdb0d7da1d23 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:17.895228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-cfafed0e-b206-4976-94ae-cdb0d7da1d23 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 23 13:44:17.895228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:17.895228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:44:17.895228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:44:17.895228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:44:17.895228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:44:17.895228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:44:17.895228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:17.895228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 23 13:44:17.895228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:17.895228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:17.895228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:17.895228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:17.895228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:17.895228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:17.895228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:17.895228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:17.896192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:17.896192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:17.896192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:17.896192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:17.896192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:17.896192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:17.896192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:44:17.896192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:44:17.896192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:44:17.896192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:44:17.896192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:17.896192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:17.896192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:17.896192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:17.896192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:17.896192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:17.897110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:17.897110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:17.897110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:17.897110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:17.897110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:17.897110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:17.897110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:17.897110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:17.897110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:17.897110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:17.897110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:44:17.897110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:17.897110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 23 13:44:17.897110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:17.897110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 459/924] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:44:17 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:44:17.902730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bbb427-0769-458e-9cf8-f6880dc5c5cd None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:17.952754 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:18.013392 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bbb427-0769-458e-9cf8-f6880dc5c5cd None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:18.049886 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec836ea-8e0a-4c01-99b7-9ae76d79126c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:18.050522 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec836ea-8e0a-4c01-99b7-9ae76d79126c None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:18.050799 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec836ea-8e0a-4c01-99b7-9ae76d79126c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:18.051071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec836ea-8e0a-4c01-99b7-9ae76d79126c None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:18.051768 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec836ea-8e0a-4c01-99b7-9ae76d79126c None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:18.052067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec836ea-8e0a-4c01-99b7-9ae76d79126c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:18.053498 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec836ea-8e0a-4c01-99b7-9ae76d79126c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:18.058876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 466/925] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:44:17 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:18.064177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfafed0e-b206-4976-94ae-cdb0d7da1d23 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:18.102421 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec836ea-8e0a-4c01-99b7-9ae76d79126c None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:18.192480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfafed0e-b206-4976-94ae-cdb0d7da1d23 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:18.236042 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba6cfa3-939b-413e-a442-9dd59a70f88a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:18.236716 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba6cfa3-939b-413e-a442-9dd59a70f88a None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:18.237186 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba6cfa3-939b-413e-a442-9dd59a70f88a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:18.237570 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba6cfa3-939b-413e-a442-9dd59a70f88a None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:18.238209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba6cfa3-939b-413e-a442-9dd59a70f88a None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:18.238537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba6cfa3-939b-413e-a442-9dd59a70f88a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:18.240170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba6cfa3-939b-413e-a442-9dd59a70f88a None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:18.245785 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 460/926] 10.222.0.62 () {56 vars in 1114 bytes} [Fri Sep 23 13:44:18 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:18.247129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec836ea-8e0a-4c01-99b7-9ae76d79126c None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:18.250943 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba6cfa3-939b-413e-a442-9dd59a70f88a None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:18.258651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:18.293298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31bf1297-ae1f-4fdf-b1d8-8957eaf7f8da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:18.293859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bf1297-ae1f-4fdf-b1d8-8957eaf7f8da None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:18.294138 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bf1297-ae1f-4fdf-b1d8-8957eaf7f8da None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:18.294437 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bf1297-ae1f-4fdf-b1d8-8957eaf7f8da None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:18.307634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31bf1297-ae1f-4fdf-b1d8-8957eaf7f8da None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:44:18.312398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-31bf1297-ae1f-4fdf-b1d8-8957eaf7f8da None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 23 13:44:18.312398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:18.312398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:18.312398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:18.312398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:18.312398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:18.312398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:18.312398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:18.312398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:18.312398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:18.312398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:18.312398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:18.312398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:44:18.312398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:44:18.312398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:44:18.312398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:44:18.313366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:44:18.313366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:44:18.313366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:44:18.313366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:44:18.313366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:18.313366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:18.313366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:18.313366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:18.313366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:18.313366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:18.313366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:18.313366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:18.313366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:18.313366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:18.313366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:18.313366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:18.314210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:18.314210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:18.314210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:18.314210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:18.314210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:44:18.314210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:44:18.314210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:44:18.314210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:44:18.314210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:44:18.314210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:44:18.314210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 23 13:44:18.314210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:18.314210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 467/927] 10.222.0.62 () {56 vars in 1120 bytes} [Fri Sep 23 13:44:18 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:44:18.316086 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c74c207-d812-40c0-82f4-ef849f5f332b None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:18.316086 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c74c207-d812-40c0-82f4-ef849f5f332b None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:18.316086 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c74c207-d812-40c0-82f4-ef849f5f332b None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:18.316763 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:18.321245 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 468/928] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:18 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:18.322062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c74c207-d812-40c0-82f4-ef849f5f332b None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:18.369957 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba6cfa3-939b-413e-a442-9dd59a70f88a None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:18.411409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8988d4e-5b76-461a-82ee-f2cb0fa7bd2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:18.412018 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8988d4e-5b76-461a-82ee-f2cb0fa7bd2c None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:18.412296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8988d4e-5b76-461a-82ee-f2cb0fa7bd2c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:18.412565 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8988d4e-5b76-461a-82ee-f2cb0fa7bd2c None admin] PATH_INFO: `/v3/endpoints` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:18.413232 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8988d4e-5b76-461a-82ee-f2cb0fa7bd2c None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:18.414636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8988d4e-5b76-461a-82ee-f2cb0fa7bd2c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:18.432768 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c74c207-d812-40c0-82f4-ef849f5f332b None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:18.465802 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8962a616-d85d-4024-84a7-8e0876460fe3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:18.466443 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8962a616-d85d-4024-84a7-8e0876460fe3 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:18.466719 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8962a616-d85d-4024-84a7-8e0876460fe3 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:18.466985 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8962a616-d85d-4024-84a7-8e0876460fe3 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:18.467680 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8962a616-d85d-4024-84a7-8e0876460fe3 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:18.467949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8962a616-d85d-4024-84a7-8e0876460fe3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:18.469290 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8962a616-d85d-4024-84a7-8e0876460fe3 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:18.480309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 469/929] 10.222.0.62 () {56 vars in 1115 bytes} [Fri Sep 23 13:44:18 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:18.481616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048de800-18cc-4a4c-bc66-2b9643e27b6f None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:18.481897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048de800-18cc-4a4c-bc66-2b9643e27b6f None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:18.482165 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048de800-18cc-4a4c-bc66-2b9643e27b6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:18.493249 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 461/930] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 13:44:18 2022] POST /identity/v3/endpoints => generated 346 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:44:18.494545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8988d4e-5b76-461a-82ee-f2cb0fa7bd2c None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:18.510163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-048de800-18cc-4a4c-bc66-2b9643e27b6f None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:18.513830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-048de800-18cc-4a4c-bc66-2b9643e27b6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:18.539858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:18.600254 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 470/931] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:18 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:18.636545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8988d4e-5b76-461a-82ee-f2cb0fa7bd2c None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:18.650070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:18.677719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4d3703-021d-466e-a4e5-d410f7da301c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:18.678803 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4d3703-021d-466e-a4e5-d410f7da301c None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:18.679461 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4d3703-021d-466e-a4e5-d410f7da301c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:18.680059 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4d3703-021d-466e-a4e5-d410f7da301c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:18.694256 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4d3703-021d-466e-a4e5-d410f7da301c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:18.696701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4d3703-021d-466e-a4e5-d410f7da301c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:18.715024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-9f4d3703-021d-466e-a4e5-d410f7da301c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:18.715024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:18.715024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:18.715024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:18.715024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:18.715024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:18.715024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:18.715024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:18.715024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:18.715024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:18.715024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:18.715024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:18.715024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:18.715024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:18.715024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:18.715024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:18.716177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:18.716177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:18.716177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:18.716177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:18.716177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:18.716177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:18.716177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:18.716177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:18.716177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:18.716177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:18.716177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:18.716177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:18.716177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:18.716177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:18.716177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:18.716177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:18.716177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:18.717429 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:18.717429 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:18.717429 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:18.717429 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 462/932] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:18.722171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048de800-18cc-4a4c-bc66-2b9643e27b6f None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:18.836929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048de800-18cc-4a4c-bc66-2b9643e27b6f None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:18.873387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f72e99-0f30-4e1c-bf4c-62e65c9434c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:18.873912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f72e99-0f30-4e1c-bf4c-62e65c9434c6 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:18.874186 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f72e99-0f30-4e1c-bf4c-62e65c9434c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:18.874487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f72e99-0f30-4e1c-bf4c-62e65c9434c6 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:18.875124 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f72e99-0f30-4e1c-bf4c-62e65c9434c6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:18.875422 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f72e99-0f30-4e1c-bf4c-62e65c9434c6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:18.876771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f72e99-0f30-4e1c-bf4c-62e65c9434c6 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:18.889401 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 471/933] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:18.895448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4d3703-021d-466e-a4e5-d410f7da301c None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:18.897541 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f72e99-0f30-4e1c-bf4c-62e65c9434c6 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:18.916315 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:19.021317 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4d3703-021d-466e-a4e5-d410f7da301c None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:19.024939 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f72e99-0f30-4e1c-bf4c-62e65c9434c6 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:19.060769 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aac918-c590-42a5-ad84-677a537a7d52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:19.061653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aac918-c590-42a5-ad84-677a537a7d52 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:19.062077 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aac918-c590-42a5-ad84-677a537a7d52 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:19.062629 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1aac918-c590-42a5-ad84-677a537a7d52 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:19.063562 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1aac918-c590-42a5-ad84-677a537a7d52 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:19.066048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1aac918-c590-42a5-ad84-677a537a7d52 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:19.066593 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb77852a-24fd-4845-bef2-cb9053283a6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:19.067254 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb77852a-24fd-4845-bef2-cb9053283a6a None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:19.067613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb77852a-24fd-4845-bef2-cb9053283a6a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:19.067951 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb77852a-24fd-4845-bef2-cb9053283a6a None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:19.076163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-e1aac918-c590-42a5-ad84-677a537a7d52 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 23 13:44:19.076163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:19.076163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:44:19.076163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:44:19.076163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:44:19.076163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:44:19.076163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:44:19.076163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:19.076163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 23 13:44:19.076163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:19.076163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:19.076163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:19.076163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:19.076163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:19.076163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:19.076163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:19.077308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:19.077308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:19.077308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:19.077308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:19.077308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:19.077308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:19.077308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:19.077308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:44:19.077308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:44:19.077308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:44:19.077308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:44:19.077308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:19.077308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:19.077308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:19.077308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:19.077308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:19.077308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:19.078531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:19.078531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:19.078531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:19.078531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:19.078531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:19.078531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:19.078531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:19.078531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:19.078531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:19.078531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:19.078531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:44:19.078531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:19.078531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 23 13:44:19.078531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:19.079455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb77852a-24fd-4845-bef2-cb9053283a6a None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:19.080562 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb77852a-24fd-4845-bef2-cb9053283a6a None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:19.081476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 472/934] 10.222.0.62 () {56 vars in 1132 bytes} [Fri Sep 23 13:44:18 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:44:19.086927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aac918-c590-42a5-ad84-677a537a7d52 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:19.099030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-fb77852a-24fd-4845-bef2-cb9053283a6a None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 23 13:44:19.099030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:19.099030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:19.099030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:19.099030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:19.099030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:19.099030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:19.099030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:19.099030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:19.099030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:19.099030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:19.099030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:19.099030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:44:19.099030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:44:19.099030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:44:19.099030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:44:19.100260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:19.100260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:19.100260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:44:19.100260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:19.100260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:44:19.100260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:19.100260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:19.100260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:19.100260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:19.100260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:19.100260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:19.100260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:19.100260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:19.100260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:19.100260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:19.100260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:19.100260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:19.100260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:19.101419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:19.101419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:19.101419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:44:19.101419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:44:19.101419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:44:19.101419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:44:19.101419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:44:19.101419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:44:19.101419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 23 13:44:19.101419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:19.104451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 463/935] 10.222.0.62 () {56 vars in 1117 bytes} [Fri Sep 23 13:44:18 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:19.106443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb77852a-24fd-4845-bef2-cb9053283a6a None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:19.136677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:19.209331 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:19.223126 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1aac918-c590-42a5-ad84-677a537a7d52 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:19.233834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb77852a-24fd-4845-bef2-cb9053283a6a None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:19.261209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d65810-5541-4771-9ea6-22666a95a407 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:19.261963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d65810-5541-4771-9ea6-22666a95a407 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:19.262339 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d65810-5541-4771-9ea6-22666a95a407 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:19.262691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d65810-5541-4771-9ea6-22666a95a407 None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:19.263452 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d65810-5541-4771-9ea6-22666a95a407 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:19.263935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d65810-5541-4771-9ea6-22666a95a407 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:19.265479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d65810-5541-4771-9ea6-22666a95a407 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:19.274497 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 473/936] 10.222.0.62 () {56 vars in 1126 bytes} [Fri Sep 23 13:44:19 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:19.280231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d65810-5541-4771-9ea6-22666a95a407 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:19.281941 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce197bea-0345-45ae-a687-1fde15353444 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:19.282682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce197bea-0345-45ae-a687-1fde15353444 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:19.283037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce197bea-0345-45ae-a687-1fde15353444 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:19.283628 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce197bea-0345-45ae-a687-1fde15353444 None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:19.284549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce197bea-0345-45ae-a687-1fde15353444 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:19.285842 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce197bea-0345-45ae-a687-1fde15353444 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:19.287365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce197bea-0345-45ae-a687-1fde15353444 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:19.315288 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 464/937] 10.222.0.62 () {56 vars in 1149 bytes} [Fri Sep 23 13:44:19 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:19.316280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce197bea-0345-45ae-a687-1fde15353444 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:19.396589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d65810-5541-4771-9ea6-22666a95a407 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:19.422140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:19.442819 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242e2fca-8775-4ac0-a1bf-5d3019756bec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:19.443438 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242e2fca-8775-4ac0-a1bf-5d3019756bec None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:19.443717 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242e2fca-8775-4ac0-a1bf-5d3019756bec None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:19.443986 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242e2fca-8775-4ac0-a1bf-5d3019756bec None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:19.444673 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242e2fca-8775-4ac0-a1bf-5d3019756bec None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:19.444974 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242e2fca-8775-4ac0-a1bf-5d3019756bec None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:19.445334 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce197bea-0345-45ae-a687-1fde15353444 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:19.446432 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242e2fca-8775-4ac0-a1bf-5d3019756bec None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:19.454486 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 474/938] 10.222.0.62 () {56 vars in 1126 bytes} [Fri Sep 23 13:44:19 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:19.459285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242e2fca-8775-4ac0-a1bf-5d3019756bec None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:19.473203 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:19.488556 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b94835-bcf7-40c9-8110-a0ef13bc44b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:19.489452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b94835-bcf7-40c9-8110-a0ef13bc44b4 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:19.489991 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b94835-bcf7-40c9-8110-a0ef13bc44b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:19.490433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b94835-bcf7-40c9-8110-a0ef13bc44b4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:19.501390 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b94835-bcf7-40c9-8110-a0ef13bc44b4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:19.503250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b94835-bcf7-40c9-8110-a0ef13bc44b4 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:19.520106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-90b94835-bcf7-40c9-8110-a0ef13bc44b4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:19.520106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:19.520106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:19.520106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:19.520106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:19.520106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:19.520106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:19.520106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:19.520106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:19.520106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:19.520106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:19.520106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:19.520106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:19.520106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:19.520106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:19.520106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:19.521277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:19.521277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:19.521277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:19.521277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:19.521277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:19.521277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:19.521277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:19.521277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:19.521277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:19.521277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:19.521277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:19.521277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:19.521277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:19.521277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:19.521277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:19.521277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:19.521277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:19.522207 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:19.522207 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:19.522207 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:19.524043 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 465/939] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:19.529939 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b94835-bcf7-40c9-8110-a0ef13bc44b4 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:19.585382 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242e2fca-8775-4ac0-a1bf-5d3019756bec None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:19.622648 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684d7ac1-9a1e-4cd2-aaed-717485c55002 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:19.623619 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684d7ac1-9a1e-4cd2-aaed-717485c55002 None admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:19.624107 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684d7ac1-9a1e-4cd2-aaed-717485c55002 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:19.624575 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684d7ac1-9a1e-4cd2-aaed-717485c55002 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:19.625451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684d7ac1-9a1e-4cd2-aaed-717485c55002 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:19.627138 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684d7ac1-9a1e-4cd2-aaed-717485c55002 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:19.639779 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b94835-bcf7-40c9-8110-a0ef13bc44b4 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:19.673313 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:19.678819 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c42999-f984-4fb3-b2ee-f06f1ef98107 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:19.679442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c42999-f984-4fb3-b2ee-f06f1ef98107 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:19.679718 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c42999-f984-4fb3-b2ee-f06f1ef98107 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:19.680611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c42999-f984-4fb3-b2ee-f06f1ef98107 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:19.680611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c42999-f984-4fb3-b2ee-f06f1ef98107 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:19.680960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c42999-f984-4fb3-b2ee-f06f1ef98107 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:19.682420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9c42999-f984-4fb3-b2ee-f06f1ef98107 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:19.700294 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 466/940] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:19.701346 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c42999-f984-4fb3-b2ee-f06f1ef98107 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:19.758359 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 475/941] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 13:44:19 2022] POST /identity/v3/endpoints => generated 347 bytes in 291 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:44:19.787874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad02529-8d0b-46bc-b429-a2b1d8c5b57b None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:19.788303 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad02529-8d0b-46bc-b429-a2b1d8c5b57b None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:19.788673 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad02529-8d0b-46bc-b429-a2b1d8c5b57b None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:19.790138 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 476/942] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:44:19.796048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1d5ea6-00ae-4d1d-8112-538d71ccbb9e None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:19.796353 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1d5ea6-00ae-4d1d-8112-538d71ccbb9e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:19.796590 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1d5ea6-00ae-4d1d-8112-538d71ccbb9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:19.817752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-4d1d5ea6-00ae-4d1d-8112-538d71ccbb9e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:19.819076 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-4d1d5ea6-00ae-4d1d-8112-538d71ccbb9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:19.826690 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c42999-f984-4fb3-b2ee-f06f1ef98107 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:19.879462 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9100635b-8a6c-4cd0-9c02-985ec0194e7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:19.880904 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9100635b-8a6c-4cd0-9c02-985ec0194e7a None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:19.881421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9100635b-8a6c-4cd0-9c02-985ec0194e7a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:19.881905 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9100635b-8a6c-4cd0-9c02-985ec0194e7a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:19.890243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:19.893635 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9100635b-8a6c-4cd0-9c02-985ec0194e7a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:19.896218 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9100635b-8a6c-4cd0-9c02-985ec0194e7a None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:19.906432 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 477/943] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:19 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 23 13:44:19.912141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-9100635b-8a6c-4cd0-9c02-985ec0194e7a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:19.912141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:19.912141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:19.912141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:19.912141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:19.912141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:19.912141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:19.912141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:19.912141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:19.912141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:19.912141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:19.912141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:19.912141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:44:19.912141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:44:19.912141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:44:19.913104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:44:19.913104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:19.913104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:19.913104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:19.913104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:19.913104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:19.913104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:19.913104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:19.913104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:19.913104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:19.913104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:19.913104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:19.913104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:19.913104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:19.913104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:19.913104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:19.913104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:19.913976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:44:19.913976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:44:19.913976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:44:19.913976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:44:19.913976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:44:19.913976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:44:19.913976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:19.913976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:19.914903 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 467/944] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:44:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:19.920448 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1d5ea6-00ae-4d1d-8112-538d71ccbb9e None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:20.036382 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1d5ea6-00ae-4d1d-8112-538d71ccbb9e None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:20.072856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242e8c73-9964-4b41-96fe-ee0bcd1d0475 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:20.073838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242e8c73-9964-4b41-96fe-ee0bcd1d0475 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:20.074319 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242e8c73-9964-4b41-96fe-ee0bcd1d0475 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:20.074884 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242e8c73-9964-4b41-96fe-ee0bcd1d0475 None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:20.075872 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242e8c73-9964-4b41-96fe-ee0bcd1d0475 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:20.076375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242e8c73-9964-4b41-96fe-ee0bcd1d0475 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:20.077994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242e8c73-9964-4b41-96fe-ee0bcd1d0475 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:20.090796 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 478/945] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:44:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:20.097740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9100635b-8a6c-4cd0-9c02-985ec0194e7a None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:20.149089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:20.187678 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242e8c73-9964-4b41-96fe-ee0bcd1d0475 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:20.203782 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9100635b-8a6c-4cd0-9c02-985ec0194e7a None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:20.242765 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f26c5d-aebe-400f-9c57-3398d5176b70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:20.243601 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f26c5d-aebe-400f-9c57-3398d5176b70 None admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:20.244036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f26c5d-aebe-400f-9c57-3398d5176b70 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:20.244408 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f26c5d-aebe-400f-9c57-3398d5176b70 None admin] PATH_INFO: `/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/b471d768cd814ff8b4f6bd35d21e24a8/roles/b2084c06d5a44297a9a7795d559fae94` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:20.246729 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f26c5d-aebe-400f-9c57-3398d5176b70 None admin] RBAC: Authorizing `identity:create_grant(project_id=2a19fd4315ad4a6eb349dfe8cfa66c23, user_id=b471d768cd814ff8b4f6bd35d21e24a8, role_id=b2084c06d5a44297a9a7795d559fae94)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:20.248493 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f26c5d-aebe-400f-9c57-3398d5176b70 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:20.265208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:20.310446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242e8c73-9964-4b41-96fe-ee0bcd1d0475 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:20.349810 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13be993d-13e7-46d0-8afa-a977ef0acd98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:20.350819 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13be993d-13e7-46d0-8afa-a977ef0acd98 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:20.351257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13be993d-13e7-46d0-8afa-a977ef0acd98 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:20.355749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13be993d-13e7-46d0-8afa-a977ef0acd98 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:20.355749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13be993d-13e7-46d0-8afa-a977ef0acd98 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:20.355977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 468/946] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:44:20 2022] PUT /identity/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/b471d768cd814ff8b4f6bd35d21e24a8/roles/b2084c06d5a44297a9a7795d559fae94 => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 13:44:20.356068 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13be993d-13e7-46d0-8afa-a977ef0acd98 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:20.365813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-13be993d-13e7-46d0-8afa-a977ef0acd98 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 23 13:44:20.365813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:20.365813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:44:20.365813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:44:20.365813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:44:20.365813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:44:20.365813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:44:20.365813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:20.365813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 23 13:44:20.365813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:20.365813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:20.365813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:20.365813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:20.365813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:20.365813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:20.365813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:20.365813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:20.367189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:20.367189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:20.367189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:20.367189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:20.367189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:20.367189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:20.367189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:44:20.367189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:44:20.367189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:44:20.367189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:44:20.367189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:20.367189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:20.367189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:20.367189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:20.367189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:20.367189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:20.368055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:20.368055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:20.368055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:20.368055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:20.368055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:20.368055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:20.368055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:20.368055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:20.368055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:20.368055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:20.368055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:44:20.368055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:20.368055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 23 13:44:20.368055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:20.368055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 479/947] 10.222.0.62 () {56 vars in 1126 bytes} [Fri Sep 23 13:44:20 2022] GET /identity/v3/services/object-store => generated 93 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:20.375783 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f26c5d-aebe-400f-9c57-3398d5176b70 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:20.504453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f26c5d-aebe-400f-9c57-3398d5176b70 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:20.515809 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:20.545731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056c6dcc-7f33-4dd9-b7b2-2358e147be55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:20.546574 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056c6dcc-7f33-4dd9-b7b2-2358e147be55 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:20.546961 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056c6dcc-7f33-4dd9-b7b2-2358e147be55 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:20.547336 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056c6dcc-7f33-4dd9-b7b2-2358e147be55 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:20.548082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056c6dcc-7f33-4dd9-b7b2-2358e147be55 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:20.548491 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056c6dcc-7f33-4dd9-b7b2-2358e147be55 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:20.551144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056c6dcc-7f33-4dd9-b7b2-2358e147be55 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:20.556982 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 469/948] 10.222.0.62 () {56 vars in 1122 bytes} [Fri Sep 23 13:44:20 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:20.565220 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13be993d-13e7-46d0-8afa-a977ef0acd98 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:20.676114 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13be993d-13e7-46d0-8afa-a977ef0acd98 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:20.694145 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:20.712817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1748c4e3-9589-4b5a-b4c2-f8121135a5a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:20.713704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1748c4e3-9589-4b5a-b4c2-f8121135a5a2 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:20.714121 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1748c4e3-9589-4b5a-b4c2-f8121135a5a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:20.714565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1748c4e3-9589-4b5a-b4c2-f8121135a5a2 None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:20.715366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1748c4e3-9589-4b5a-b4c2-f8121135a5a2 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:20.715806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1748c4e3-9589-4b5a-b4c2-f8121135a5a2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:20.717367 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1748c4e3-9589-4b5a-b4c2-f8121135a5a2 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:20.723637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 480/949] 10.222.0.62 () {56 vars in 1122 bytes} [Fri Sep 23 13:44:20 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:20.730566 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056c6dcc-7f33-4dd9-b7b2-2358e147be55 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:20.841629 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056c6dcc-7f33-4dd9-b7b2-2358e147be55 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:20.877058 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e99b2f-4053-4ca4-bab6-29475339d0ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:20.878063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e99b2f-4053-4ca4-bab6-29475339d0ee None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:20.878506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e99b2f-4053-4ca4-bab6-29475339d0ee None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:20.878980 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e99b2f-4053-4ca4-bab6-29475339d0ee None admin] PATH_INFO: `/v3/endpoints` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:20.879821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e99b2f-4053-4ca4-bab6-29475339d0ee None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:20.881366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e99b2f-4053-4ca4-bab6-29475339d0ee None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:20.912957 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600ef60d-812f-4d48-b49b-dd6d1d54d532 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:20.913236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600ef60d-812f-4d48-b49b-dd6d1d54d532 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:20.913507 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600ef60d-812f-4d48-b49b-dd6d1d54d532 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:20.915087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 481/950] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:44:20.919949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bfa70d-72bc-4d9d-94ef-865e40ca956e None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:20.920225 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bfa70d-72bc-4d9d-94ef-865e40ca956e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:20.920491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bfa70d-72bc-4d9d-94ef-865e40ca956e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:20.940856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c0bfa70d-72bc-4d9d-94ef-865e40ca956e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:20.942098 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c0bfa70d-72bc-4d9d-94ef-865e40ca956e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:20.988526 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 470/951] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 13:44:20 2022] POST /identity/v3/endpoints => generated 349 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 13:44:21.038415 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 482/952] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:20 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 23 13:44:21.316370 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e99b2f-4053-4ca4-bab6-29475339d0ee None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:21.353081 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:21.429854 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e99b2f-4053-4ca4-bab6-29475339d0ee None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:21.464784 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f793830c-dba0-4dc5-98d3-d2ac98765eca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:21.465321 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f793830c-dba0-4dc5-98d3-d2ac98765eca None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:21.465595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f793830c-dba0-4dc5-98d3-d2ac98765eca None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:21.465865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f793830c-dba0-4dc5-98d3-d2ac98765eca None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:21.466534 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f793830c-dba0-4dc5-98d3-d2ac98765eca None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:21.467947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f793830c-dba0-4dc5-98d3-d2ac98765eca None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:21.470640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: WARNING keystone.server.flask.common [None req-f793830c-dba0-4dc5-98d3-d2ac98765eca None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 23 13:44:21.513605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.sql.core [None req-f793830c-dba0-4dc5-98d3-d2ac98765eca None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 23 13:44:21.513605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 23 13:44:21.513605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [parameters: {'id': 'fcd3f83711a546ff9a39e40c12a48b2b', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 23 13:44:21.513605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81569) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 23 13:44:21.521029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-f793830c-dba0-4dc5-98d3-d2ac98765eca None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id fcd3f83711a546ff9a39e40c12a48b2b.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id fcd3f83711a546ff9a39e40c12a48b2b. Sep 23 13:44:21.521029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:21.521029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 23 13:44:21.521029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 23 13:44:21.521029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 23 13:44:21.521029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 23 13:44:21.521029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 23 13:44:21.521029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application result = self._query(query) Sep 23 13:44:21.521029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 23 13:44:21.521029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application conn.query(q) Sep 23 13:44:21.521029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 23 13:44:21.521029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 23 13:44:21.521029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 23 13:44:21.528731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application result.read() Sep 23 13:44:21.528731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 23 13:44:21.528731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 23 13:44:21.528731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 23 13:44:21.528731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 23 13:44:21.528731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 23 13:44:21.528731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 23 13:44:21.528731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 23 13:44:21.528731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 23 13:44:21.528731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 23 13:44:21.528731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:21.528731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 23 13:44:21.528731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:21.528731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:21.528731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 23 13:44:21.528731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 23 13:44:21.528731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 23 13:44:21.528731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 23 13:44:21.529676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 23 13:44:21.529676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application next(self.gen) Sep 23 13:44:21.529676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 23 13:44:21.529676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application yield resource Sep 23 13:44:21.529676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 23 13:44:21.529676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application next(self.gen) Sep 23 13:44:21.529676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 23 13:44:21.529676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self.session.rollback() Sep 23 13:44:21.529676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 23 13:44:21.529676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self.force_reraise() Sep 23 13:44:21.529676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 23 13:44:21.529676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise self.value Sep 23 13:44:21.529676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 23 13:44:21.529676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 23 13:44:21.529676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 23 13:44:21.529676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application session.commit() Sep 23 13:44:21.529676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 23 13:44:21.530588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 23 13:44:21.530588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 23 13:44:21.530588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._prepare_impl() Sep 23 13:44:21.530588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 23 13:44:21.530588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self.session.flush() Sep 23 13:44:21.530588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 23 13:44:21.530588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._flush(objects) Sep 23 13:44:21.530588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 23 13:44:21.530588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 23 13:44:21.530588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 23 13:44:21.530588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application compat.raise_( Sep 23 13:44:21.530588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 23 13:44:21.530588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception Sep 23 13:44:21.530588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 23 13:44:21.530588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application flush_context.execute() Sep 23 13:44:21.530588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 23 13:44:21.530588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rec.execute(self) Sep 23 13:44:21.531451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 23 13:44:21.531451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 23 13:44:21.531451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 23 13:44:21.531451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 23 13:44:21.531451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 23 13:44:21.531451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 23 13:44:21.531451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 23 13:44:21.531451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 23 13:44:21.531451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 23 13:44:21.531451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 23 13:44:21.531451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 23 13:44:21.531451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 23 13:44:21.531451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 23 13:44:21.531451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 23 13:44:21.531451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 23 13:44:21.532228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 23 13:44:21.532228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 23 13:44:21.532228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception Sep 23 13:44:21.532228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 23 13:44:21.532228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 23 13:44:21.532228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 23 13:44:21.532228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 23 13:44:21.532228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 23 13:44:21.532228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application result = self._query(query) Sep 23 13:44:21.532228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 23 13:44:21.532228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application conn.query(q) Sep 23 13:44:21.532228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 23 13:44:21.532228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 23 13:44:21.532228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 23 13:44:21.532228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application result.read() Sep 23 13:44:21.532228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 23 13:44:21.532228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 23 13:44:21.533682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 23 13:44:21.533682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 23 13:44:21.533682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 23 13:44:21.533682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 23 13:44:21.533682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 23 13:44:21.533682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 23 13:44:21.533682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 23 13:44:21.533682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 23 13:44:21.533682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [parameters: {'id': 'fcd3f83711a546ff9a39e40c12a48b2b', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 23 13:44:21.533682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 23 13:44:21.533682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:21.533682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:21.533682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:21.533682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:21.533682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 23 13:44:21.536538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 23 13:44:21.536538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 23 13:44:21.536538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 23 13:44:21.536538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 23 13:44:21.536538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:21.536538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:21.536538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:21.536538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:21.536538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:21.536538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:21.536538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:21.536538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:21.536538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:21.536538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:21.536538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:21.536538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:21.536538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:21.539696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:21.539696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 23 13:44:21.539696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 23 13:44:21.539696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:21.539696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:21.539696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 23 13:44:21.539696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 23 13:44:21.539696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id fcd3f83711a546ff9a39e40c12a48b2b. Sep 23 13:44:21.539696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:21.539696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 471/953] 10.222.0.62 () {60 vars in 1140 bytes} [Fri Sep 23 13:44:21 2022] POST /identity/v3/projects => generated 262 bytes in 207 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:21.541002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0bfa70d-72bc-4d9d-94ef-865e40ca956e None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:21.647594 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0bfa70d-72bc-4d9d-94ef-865e40ca956e None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:21.660359 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:21.689862 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa8a9cd-bfe7-4cff-9b1f-82f6c4e1e0a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:21.690522 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa8a9cd-bfe7-4cff-9b1f-82f6c4e1e0a9 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:21.690797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa8a9cd-bfe7-4cff-9b1f-82f6c4e1e0a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:21.691065 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa8a9cd-bfe7-4cff-9b1f-82f6c4e1e0a9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:21.698823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa8a9cd-bfe7-4cff-9b1f-82f6c4e1e0a9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:21.700265 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa8a9cd-bfe7-4cff-9b1f-82f6c4e1e0a9 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:21.712272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-6aa8a9cd-bfe7-4cff-9b1f-82f6c4e1e0a9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:21.712272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:21.712272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:21.712272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:21.712272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:21.712272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:21.712272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:21.712272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:21.712272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:21.712272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:21.712272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:21.712272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:21.712272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:44:21.712272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:44:21.712272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:44:21.713297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:44:21.713297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:21.713297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:21.713297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:21.713297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:21.713297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:21.713297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:21.713297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:21.713297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:21.713297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:21.713297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:21.713297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:21.713297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:21.713297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:21.713297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:21.713297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:21.713297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:21.714238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:44:21.714238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:44:21.714238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:44:21.714238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:44:21.714238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:44:21.714238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:44:21.714238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:21.714238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:21.714238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 483/954] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:44:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:21.721869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f793830c-dba0-4dc5-98d3-d2ac98765eca None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:21.811899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:21.827867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f793830c-dba0-4dc5-98d3-d2ac98765eca None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:21.863767 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0283a6f1-745b-431f-8b7f-9e68e3d057bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:21.864381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0283a6f1-745b-431f-8b7f-9e68e3d057bd None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:21.865070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0283a6f1-745b-431f-8b7f-9e68e3d057bd None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:21.865357 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0283a6f1-745b-431f-8b7f-9e68e3d057bd None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:21.866349 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0283a6f1-745b-431f-8b7f-9e68e3d057bd None admin] RBAC: Adding query filter params (name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:21.867221 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0283a6f1-745b-431f-8b7f-9e68e3d057bd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:21.868862 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0283a6f1-745b-431f-8b7f-9e68e3d057bd None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:21.890302 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 472/955] 10.222.0.62 () {56 vars in 1112 bytes} [Fri Sep 23 13:44:21 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:22.150729 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9fccdf-38d0-4164-a418-ae889f9cd1be None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:22.151216 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9fccdf-38d0-4164-a418-ae889f9cd1be None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:22.151216 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9fccdf-38d0-4164-a418-ae889f9cd1be None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:22.152887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 484/956] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:22.158620 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f41662-f76d-4378-b0ff-45ebfde1ad5b None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:22.158879 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f41662-f76d-4378-b0ff-45ebfde1ad5b None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:22.159162 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f41662-f76d-4378-b0ff-45ebfde1ad5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:22.178894 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-53f41662-f76d-4378-b0ff-45ebfde1ad5b None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:22.180412 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-53f41662-f76d-4378-b0ff-45ebfde1ad5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:22.258447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 473/957] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:22 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:22.523901 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9fccdf-38d0-4164-a418-ae889f9cd1be None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:22.642833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9fccdf-38d0-4164-a418-ae889f9cd1be None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:22.660105 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12004a39-b289-4fc7-a6a7-63212f99adc5 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:22.661033 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12004a39-b289-4fc7-a6a7-63212f99adc5 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:22.661640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12004a39-b289-4fc7-a6a7-63212f99adc5 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:22.663870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 474/958] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:22 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:22.671013 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e78167c-f15a-4ba3-92e7-2a35772cefbf None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:22.671301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e78167c-f15a-4ba3-92e7-2a35772cefbf None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:22.671582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e78167c-f15a-4ba3-92e7-2a35772cefbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:22.681900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec3d1b6-ad04-4dac-9ef9-b330aea690f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:22.682924 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec3d1b6-ad04-4dac-9ef9-b330aea690f1 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:22.683401 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec3d1b6-ad04-4dac-9ef9-b330aea690f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:22.683863 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec3d1b6-ad04-4dac-9ef9-b330aea690f1 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:22.684638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec3d1b6-ad04-4dac-9ef9-b330aea690f1 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:22.686309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec3d1b6-ad04-4dac-9ef9-b330aea690f1 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:22.692550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-8e78167c-f15a-4ba3-92e7-2a35772cefbf None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:22.693749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8e78167c-f15a-4ba3-92e7-2a35772cefbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:22.696434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-cec3d1b6-ad04-4dac-9ef9-b330aea690f1 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 23 13:44:22.696434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:22.696434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:44:22.696434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:44:22.696434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:44:22.696434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:44:22.696434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:44:22.696434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:22.696434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 23 13:44:22.696434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:22.696434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:22.696434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:22.696434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:22.696434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:22.696434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:22.696434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:22.696434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:22.697442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:22.697442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:22.697442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:22.697442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:22.697442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:22.697442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:22.697442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:44:22.697442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:44:22.697442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:44:22.697442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:44:22.697442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:22.697442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:22.697442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:22.697442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:22.697442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:22.697442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:22.698311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:22.698311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:22.698311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:22.698311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:22.698311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:22.698311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:22.698311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:22.698311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:22.698311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:22.698311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:22.698311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:44:22.698311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:22.698311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 23 13:44:22.698311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:22.701089 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 485/959] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:44:22 2022] GET /identity/v3/services/compute => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:22.708340 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec3d1b6-ad04-4dac-9ef9-b330aea690f1 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:22.742931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:22.779892 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 475/960] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:22 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:22.823085 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec3d1b6-ad04-4dac-9ef9-b330aea690f1 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:22.858820 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb3b92d-8371-4579-bc52-6d7684cfa89c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:22.859783 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb3b92d-8371-4579-bc52-6d7684cfa89c None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:22.860258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb3b92d-8371-4579-bc52-6d7684cfa89c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:22.860720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb3b92d-8371-4579-bc52-6d7684cfa89c None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:22.862366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb3b92d-8371-4579-bc52-6d7684cfa89c None admin] RBAC: Adding query filter params (name=compute) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:22.862893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb3b92d-8371-4579-bc52-6d7684cfa89c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:22.864481 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb3b92d-8371-4579-bc52-6d7684cfa89c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:22.870959 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 486/961] 10.222.0.62 () {56 vars in 1112 bytes} [Fri Sep 23 13:44:22 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:22.877868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e78167c-f15a-4ba3-92e7-2a35772cefbf None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:22.881947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:22.985175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e78167c-f15a-4ba3-92e7-2a35772cefbf None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:23.027962 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d281e17-87fe-4580-9f2b-cfcf9ad5d033 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:23.028574 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d281e17-87fe-4580-9f2b-cfcf9ad5d033 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:23.028853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d281e17-87fe-4580-9f2b-cfcf9ad5d033 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:23.029121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d281e17-87fe-4580-9f2b-cfcf9ad5d033 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:23.029753 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d281e17-87fe-4580-9f2b-cfcf9ad5d033 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:23.030107 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d281e17-87fe-4580-9f2b-cfcf9ad5d033 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:23.031446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d281e17-87fe-4580-9f2b-cfcf9ad5d033 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:23.036812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 476/962] 10.222.0.62 () {56 vars in 1112 bytes} [Fri Sep 23 13:44:22 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:23.055728 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb3b92d-8371-4579-bc52-6d7684cfa89c None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:23.167649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:23.175430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb3b92d-8371-4579-bc52-6d7684cfa89c None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:23.198170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3b3935-60a3-421d-a0d5-2c3573a15f25 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:23.198514 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3b3935-60a3-421d-a0d5-2c3573a15f25 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:23.198828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3b3935-60a3-421d-a0d5-2c3573a15f25 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:23.200355 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 477/963] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:23.212466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a9c64d-9d11-4a78-bc24-c299397de26c None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:23.212466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a9c64d-9d11-4a78-bc24-c299397de26c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:23.212713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a9c64d-9d11-4a78-bc24-c299397de26c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:23.230992 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77fbbc7-1ca7-4e58-90be-098cb0fa5b7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:23.232020 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77fbbc7-1ca7-4e58-90be-098cb0fa5b7a None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:23.232498 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77fbbc7-1ca7-4e58-90be-098cb0fa5b7a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:23.232977 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77fbbc7-1ca7-4e58-90be-098cb0fa5b7a None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:23.233401 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-40a9c64d-9d11-4a78-bc24-c299397de26c None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:23.234927 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-40a9c64d-9d11-4a78-bc24-c299397de26c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:23.246809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77fbbc7-1ca7-4e58-90be-098cb0fa5b7a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:44:23.252120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-f77fbbc7-1ca7-4e58-90be-098cb0fa5b7a None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 23 13:44:23.252120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:23.252120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:23.252120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:23.252120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:23.252120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:23.252120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:23.252120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:23.252120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:23.252120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:23.252120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:23.252120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:23.252120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:44:23.252120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:44:23.252120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:44:23.252120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:44:23.253062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:44:23.253062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:44:23.253062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:44:23.253062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:44:23.253062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:23.253062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:23.253062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:23.253062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:23.253062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:23.253062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:23.253062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:23.253062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:23.253062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:23.253062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:23.253062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:23.253062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:23.253923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:23.253923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:23.253923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:23.253923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:23.253923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:44:23.253923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:44:23.253923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:44:23.253923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:44:23.253923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:44:23.253923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:44:23.253923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 23 13:44:23.253923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:23.253923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 487/964] 10.222.0.62 () {56 vars in 1120 bytes} [Fri Sep 23 13:44:23 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:23.260656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77fbbc7-1ca7-4e58-90be-098cb0fa5b7a None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:23.318844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:23.324019 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 478/965] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:23 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:23.372154 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77fbbc7-1ca7-4e58-90be-098cb0fa5b7a None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:23.407742 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44b323c-8801-4e13-b44a-73a808c7c8ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:23.408705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44b323c-8801-4e13-b44a-73a808c7c8ff None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:23.409238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44b323c-8801-4e13-b44a-73a808c7c8ff None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:23.409736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44b323c-8801-4e13-b44a-73a808c7c8ff None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:23.410696 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44b323c-8801-4e13-b44a-73a808c7c8ff None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:23.411197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44b323c-8801-4e13-b44a-73a808c7c8ff None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:23.412749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e44b323c-8801-4e13-b44a-73a808c7c8ff None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:23.424883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 488/966] 10.222.0.62 () {56 vars in 1115 bytes} [Fri Sep 23 13:44:23 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:23.430563 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a9c64d-9d11-4a78-bc24-c299397de26c None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:23.536958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a9c64d-9d11-4a78-bc24-c299397de26c None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:23.572145 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c059c156-f144-4772-998d-a26a3156099c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:23.572696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c059c156-f144-4772-998d-a26a3156099c None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:23.572970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c059c156-f144-4772-998d-a26a3156099c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:23.573238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c059c156-f144-4772-998d-a26a3156099c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:23.584054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c059c156-f144-4772-998d-a26a3156099c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:23.585517 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c059c156-f144-4772-998d-a26a3156099c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:23.602592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-c059c156-f144-4772-998d-a26a3156099c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:23.602592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:23.602592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:23.602592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:23.602592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:23.602592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:23.602592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:23.602592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:23.602592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:23.602592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:23.602592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:23.602592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:23.602592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:23.602592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:23.602592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:23.602592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:23.603564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:23.603564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:23.603564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:23.603564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:23.603564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:23.603564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:23.603564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:23.603564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:23.603564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:23.603564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:23.603564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:23.603564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:23.603564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:23.603564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:23.603564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:23.603564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:23.603564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:23.604608 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:23.604608 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:23.604608 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:23.604608 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 479/967] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:23.611262 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44b323c-8801-4e13-b44a-73a808c7c8ff None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:23.615325 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c059c156-f144-4772-998d-a26a3156099c None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:23.634052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:23.676164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:23.732936 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c059c156-f144-4772-998d-a26a3156099c None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:23.733874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e44b323c-8801-4e13-b44a-73a808c7c8ff None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:23.771978 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08287018-4750-4e92-be9d-e8b01e3467bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:23.772328 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95e34dd-3dd7-4f1a-8090-dc75c66edea7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:23.772586 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08287018-4750-4e92-be9d-e8b01e3467bd None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:23.772865 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08287018-4750-4e92-be9d-e8b01e3467bd None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:23.773133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08287018-4750-4e92-be9d-e8b01e3467bd None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:23.773780 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08287018-4750-4e92-be9d-e8b01e3467bd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:23.774023 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95e34dd-3dd7-4f1a-8090-dc75c66edea7 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:23.774137 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08287018-4750-4e92-be9d-e8b01e3467bd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:23.774630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95e34dd-3dd7-4f1a-8090-dc75c66edea7 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:23.775138 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95e34dd-3dd7-4f1a-8090-dc75c66edea7 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:23.775529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08287018-4750-4e92-be9d-e8b01e3467bd None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:23.776203 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95e34dd-3dd7-4f1a-8090-dc75c66edea7 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:23.778002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95e34dd-3dd7-4f1a-8090-dc75c66edea7 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:23.792029 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 489/968] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:23.792206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-b95e34dd-3dd7-4f1a-8090-dc75c66edea7 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 23 13:44:23.792206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:23.792206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:44:23.792206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:44:23.792206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:44:23.792206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:44:23.792206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:44:23.792206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:23.792206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 23 13:44:23.792206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:23.792206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:23.792206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:23.792206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:23.792206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:23.792206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:23.792206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:23.792206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:23.793622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:23.793622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:23.793622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:23.793622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:23.793622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:23.793622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:23.793622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:44:23.793622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:44:23.793622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:44:23.793622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:44:23.793622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:23.793622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:23.793622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:23.793622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:23.793622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:23.793622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:23.794499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:23.794499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:23.794499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:23.794499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:23.794499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:23.794499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:23.794499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:23.794499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:23.794499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:23.794499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:23.794499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:44:23.794499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:23.794499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 23 13:44:23.794499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:23.797559 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08287018-4750-4e92-be9d-e8b01e3467bd None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:23.797989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 480/969] 10.222.0.62 () {56 vars in 1126 bytes} [Fri Sep 23 13:44:23 2022] GET /identity/v3/services/object-store => generated 93 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:23.798854 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95e34dd-3dd7-4f1a-8090-dc75c66edea7 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:23.885358 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:23.909206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08287018-4750-4e92-be9d-e8b01e3467bd None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:23.911413 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95e34dd-3dd7-4f1a-8090-dc75c66edea7 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:23.927443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:23.947778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9271e86b-3cf9-491e-a81d-8a089764921f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:23.948710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9271e86b-3cf9-491e-a81d-8a089764921f None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:23.949128 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9271e86b-3cf9-491e-a81d-8a089764921f None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:23.949531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9271e86b-3cf9-491e-a81d-8a089764921f None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:23.955737 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097c1773-371d-48a6-906e-ce866e7c674c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:23.956428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097c1773-371d-48a6-906e-ce866e7c674c None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:23.956776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097c1773-371d-48a6-906e-ce866e7c674c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:23.957137 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097c1773-371d-48a6-906e-ce866e7c674c None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:23.957837 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097c1773-371d-48a6-906e-ce866e7c674c None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:23.958265 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097c1773-371d-48a6-906e-ce866e7c674c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:23.959645 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9271e86b-3cf9-491e-a81d-8a089764921f None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:23.959755 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097c1773-371d-48a6-906e-ce866e7c674c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:23.961468 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9271e86b-3cf9-491e-a81d-8a089764921f None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:23.965592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 481/970] 10.222.0.62 () {56 vars in 1122 bytes} [Fri Sep 23 13:44:23 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:23.975315 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097c1773-371d-48a6-906e-ce866e7c674c None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:23.976352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-9271e86b-3cf9-491e-a81d-8a089764921f None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 23 13:44:23.976352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:23.976352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:23.976352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:23.976352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:23.976352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:23.976352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:23.976352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:23.976352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:23.976352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:23.976352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:23.976352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:23.976352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:44:23.976352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:44:23.976352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:44:23.976352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:44:23.977258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:23.977258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:23.977258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:44:23.977258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:23.977258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:44:23.977258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:23.977258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:23.977258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:23.977258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:23.977258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:23.977258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:23.977258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:23.977258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:23.977258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:23.977258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:23.977258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:23.977258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:23.977258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:23.978223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:23.978223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:23.978223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:44:23.978223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:44:23.978223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:44:23.978223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:44:23.978223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:44:23.978223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:44:23.978223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 23 13:44:23.978223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:23.980060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 490/971] 10.222.0.62 () {56 vars in 1117 bytes} [Fri Sep 23 13:44:23 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:23.990388 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9271e86b-3cf9-491e-a81d-8a089764921f None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:24.107131 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097c1773-371d-48a6-906e-ce866e7c674c None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:24.117664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9271e86b-3cf9-491e-a81d-8a089764921f None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:24.154613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878d80ab-9df7-49c8-98a3-b061b57eca78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:24.154801 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec43c69-f30a-49a0-80d9-d50a2e1aa3ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:24.155140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878d80ab-9df7-49c8-98a3-b061b57eca78 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:24.155246 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec43c69-f30a-49a0-80d9-d50a2e1aa3ba None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:24.155415 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec43c69-f30a-49a0-80d9-d50a2e1aa3ba None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:24.155512 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878d80ab-9df7-49c8-98a3-b061b57eca78 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:24.155680 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878d80ab-9df7-49c8-98a3-b061b57eca78 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:24.155771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec43c69-f30a-49a0-80d9-d50a2e1aa3ba None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:24.156293 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878d80ab-9df7-49c8-98a3-b061b57eca78 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:24.156402 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec43c69-f30a-49a0-80d9-d50a2e1aa3ba None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:24.156587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878d80ab-9df7-49c8-98a3-b061b57eca78 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:24.156683 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec43c69-f30a-49a0-80d9-d50a2e1aa3ba None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:24.158268 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878d80ab-9df7-49c8-98a3-b061b57eca78 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:24.158818 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec43c69-f30a-49a0-80d9-d50a2e1aa3ba None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:24.169429 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:24.172112 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 482/972] 10.222.0.62 () {56 vars in 1122 bytes} [Fri Sep 23 13:44:23 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:24.178386 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878d80ab-9df7-49c8-98a3-b061b57eca78 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:24.186387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 491/973] 10.222.0.62 () {56 vars in 1149 bytes} [Fri Sep 23 13:44:23 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:24.190662 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec43c69-f30a-49a0-80d9-d50a2e1aa3ba None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:24.229561 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:24.296116 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878d80ab-9df7-49c8-98a3-b061b57eca78 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:24.336813 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e555705e-57d9-4961-9e1a-14790eb1f463 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:24.337803 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e555705e-57d9-4961-9e1a-14790eb1f463 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:24.338298 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e555705e-57d9-4961-9e1a-14790eb1f463 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:24.338846 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e555705e-57d9-4961-9e1a-14790eb1f463 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:24.339020 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec43c69-f30a-49a0-80d9-d50a2e1aa3ba None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:24.340137 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e555705e-57d9-4961-9e1a-14790eb1f463 None admin] RBAC: Adding query filter params (interface=admin, service_id=f6cccd6a7e1b4c8499a00c81c7d2a9ef, region_id=RegionOne) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:24.340710 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e555705e-57d9-4961-9e1a-14790eb1f463 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:24.342885 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e555705e-57d9-4961-9e1a-14790eb1f463 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:24.349172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 483/974] 10.222.0.62 () {56 vars in 1249 bytes} [Fri Sep 23 13:44:24 2022] GET /identity/v3/endpoints?service_id=f6cccd6a7e1b4c8499a00c81c7d2a9ef&interface=admin®ion_id=RegionOne => generated 193 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:24.350794 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe06c21-9873-482a-9552-c5617441007e None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:24.351145 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe06c21-9873-482a-9552-c5617441007e None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:24.351493 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe06c21-9873-482a-9552-c5617441007e None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:24.352946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 484/975] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:24.355032 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe06c21-9873-482a-9552-c5617441007e None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:24.383956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac79bf1a-e35c-40d1-8f5a-7f4b27c3b44c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:24.384567 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac79bf1a-e35c-40d1-8f5a-7f4b27c3b44c None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:24.384834 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac79bf1a-e35c-40d1-8f5a-7f4b27c3b44c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:24.385096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac79bf1a-e35c-40d1-8f5a-7f4b27c3b44c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:24.448592 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:24.455529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac79bf1a-e35c-40d1-8f5a-7f4b27c3b44c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:24.456983 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac79bf1a-e35c-40d1-8f5a-7f4b27c3b44c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:24.468790 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe06c21-9873-482a-9552-c5617441007e None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:24.477805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:24.506681 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7b572d-5fb0-4da6-8625-7508becfe9e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:24.507402 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7b572d-5fb0-4da6-8625-7508becfe9e5 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:24.507681 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7b572d-5fb0-4da6-8625-7508becfe9e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:24.508068 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7b572d-5fb0-4da6-8625-7508becfe9e5 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:24.508673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7b572d-5fb0-4da6-8625-7508becfe9e5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:24.510132 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7b572d-5fb0-4da6-8625-7508becfe9e5 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:24.517705 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 485/976] 10.222.0.62 () {56 vars in 1087 bytes} [Fri Sep 23 13:44:24 2022] GET /identity/v3/services => generated 1497 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 13:44:24.519032 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c134b2-8738-464e-9af3-f2ac74b2e161 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:24.519315 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c134b2-8738-464e-9af3-f2ac74b2e161 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:24.519587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c134b2-8738-464e-9af3-f2ac74b2e161 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:24.545654 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-99c134b2-8738-464e-9af3-f2ac74b2e161 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:24.546925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-99c134b2-8738-464e-9af3-f2ac74b2e161 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:24.579632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 492/977] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:44:24 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 23 13:44:24.587813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac79bf1a-e35c-40d1-8f5a-7f4b27c3b44c None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:24.650487 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 486/978] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:24 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:24.718554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:24.728234 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac79bf1a-e35c-40d1-8f5a-7f4b27c3b44c None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:24.770477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70684803-08b7-4108-849b-09ac50290940 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:24.771119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70684803-08b7-4108-849b-09ac50290940 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:24.771398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70684803-08b7-4108-849b-09ac50290940 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:24.771670 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70684803-08b7-4108-849b-09ac50290940 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:24.783354 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70684803-08b7-4108-849b-09ac50290940 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:24.785477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70684803-08b7-4108-849b-09ac50290940 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:24.801831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-70684803-08b7-4108-849b-09ac50290940 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:24.801831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:24.801831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:24.801831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:24.801831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:24.801831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:24.801831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:24.801831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:24.801831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:24.801831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:24.801831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:24.801831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:24.801831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:24.801831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:24.801831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:24.801831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:24.802846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:24.802846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:24.802846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:24.802846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:24.802846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:24.802846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:24.802846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:24.802846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:24.802846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:24.802846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:24.802846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:24.802846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:24.802846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:24.802846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:24.802846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:24.802846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:24.802846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:24.803769 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:24.803769 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:24.803769 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:24.803769 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 493/979] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:24.811396 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c134b2-8738-464e-9af3-f2ac74b2e161 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:24.914367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:24.917142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c134b2-8738-464e-9af3-f2ac74b2e161 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:24.949920 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70684803-08b7-4108-849b-09ac50290940 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:24.953901 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14e1261-b7a8-42fa-bf78-4a8bb1d8a099 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:24.955157 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14e1261-b7a8-42fa-bf78-4a8bb1d8a099 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:24.955597 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14e1261-b7a8-42fa-bf78-4a8bb1d8a099 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:24.955900 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14e1261-b7a8-42fa-bf78-4a8bb1d8a099 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:24.956551 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14e1261-b7a8-42fa-bf78-4a8bb1d8a099 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:24.956853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14e1261-b7a8-42fa-bf78-4a8bb1d8a099 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:24.958216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14e1261-b7a8-42fa-bf78-4a8bb1d8a099 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:24.976038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 487/980] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:24.977417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14e1261-b7a8-42fa-bf78-4a8bb1d8a099 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:25.066188 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70684803-08b7-4108-849b-09ac50290940 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:25.087285 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14e1261-b7a8-42fa-bf78-4a8bb1d8a099 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:25.104875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2394a4f2-5868-4b90-bd68-abd9b66f8a24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:25.105491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2394a4f2-5868-4b90-bd68-abd9b66f8a24 None admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:25.105773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2394a4f2-5868-4b90-bd68-abd9b66f8a24 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:25.106047 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2394a4f2-5868-4b90-bd68-abd9b66f8a24 None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:25.106735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2394a4f2-5868-4b90-bd68-abd9b66f8a24 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:25.108753 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2394a4f2-5868-4b90-bd68-abd9b66f8a24 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:25.113274 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: WARNING keystone.server.flask.common [None req-2394a4f2-5868-4b90-bd68-abd9b66f8a24 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 23 13:44:25.123365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc90364-a381-4db5-a40f-af9d2c3cdc67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:25.132583 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc90364-a381-4db5-a40f-af9d2c3cdc67 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:25.132583 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc90364-a381-4db5-a40f-af9d2c3cdc67 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:25.132583 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc90364-a381-4db5-a40f-af9d2c3cdc67 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:25.133718 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc90364-a381-4db5-a40f-af9d2c3cdc67 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:25.135311 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc90364-a381-4db5-a40f-af9d2c3cdc67 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:25.147432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-4cc90364-a381-4db5-a40f-af9d2c3cdc67 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:25.147432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:25.147432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:25.147432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:25.147432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:25.147432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:25.147432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:25.147432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:25.147432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:25.147432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:25.147432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:25.147432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:25.147432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:44:25.147432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:44:25.147432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:44:25.148369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:44:25.148369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:25.148369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:25.148369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:25.148369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:25.148369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:25.148369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:25.148369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:25.148369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:25.148369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:25.148369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:25.148369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:25.148369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:25.148369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:25.148369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:25.148369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:25.148369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:25.149324 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:44:25.149324 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:44:25.149324 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:44:25.149324 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:44:25.149324 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:44:25.149324 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:44:25.149324 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:25.149324 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:25.153873 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 488/981] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:44:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:25.156212 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc90364-a381-4db5-a40f-af9d2c3cdc67 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:25.174862 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.sql.core [None req-2394a4f2-5868-4b90-bd68-abd9b66f8a24 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 23 13:44:25.174862 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 23 13:44:25.174862 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [parameters: {'user_id': '4c88fb5d45884fa68c7104c2b753f0ab', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 23 13:44:25.174862 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81568) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 23 13:44:25.182075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:25.193036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-2394a4f2-5868-4b90-bd68-abd9b66f8a24 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 23 13:44:25.193036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:25.193036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 23 13:44:25.193036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 23 13:44:25.193036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 23 13:44:25.193036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 23 13:44:25.193036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 23 13:44:25.193036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application result = self._query(query) Sep 23 13:44:25.193036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 23 13:44:25.193036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application conn.query(q) Sep 23 13:44:25.193036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 23 13:44:25.193036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 23 13:44:25.193036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 23 13:44:25.193036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application result.read() Sep 23 13:44:25.193036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 23 13:44:25.194018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 23 13:44:25.194018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 23 13:44:25.194018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 23 13:44:25.194018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 23 13:44:25.194018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 23 13:44:25.194018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 23 13:44:25.194018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 23 13:44:25.194018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 23 13:44:25.194018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:25.194018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 23 13:44:25.194018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:25.194018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:25.194018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 23 13:44:25.194018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 23 13:44:25.194018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 23 13:44:25.194018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 23 13:44:25.194018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 23 13:44:25.194018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application next(self.gen) Sep 23 13:44:25.195040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 23 13:44:25.195040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application yield resource Sep 23 13:44:25.195040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 23 13:44:25.195040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application next(self.gen) Sep 23 13:44:25.195040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 23 13:44:25.195040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self.session.rollback() Sep 23 13:44:25.195040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 23 13:44:25.195040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self.force_reraise() Sep 23 13:44:25.195040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 23 13:44:25.195040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise self.value Sep 23 13:44:25.195040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 23 13:44:25.195040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 23 13:44:25.195040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 23 13:44:25.195040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application session.commit() Sep 23 13:44:25.195040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 23 13:44:25.195040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 23 13:44:25.195040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 23 13:44:25.196494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._prepare_impl() Sep 23 13:44:25.196494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 23 13:44:25.196494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self.session.flush() Sep 23 13:44:25.196494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 23 13:44:25.196494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._flush(objects) Sep 23 13:44:25.196494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 23 13:44:25.196494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 23 13:44:25.196494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 23 13:44:25.196494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application compat.raise_( Sep 23 13:44:25.196494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 23 13:44:25.196494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception Sep 23 13:44:25.196494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 23 13:44:25.196494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application flush_context.execute() Sep 23 13:44:25.196494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 23 13:44:25.196494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rec.execute(self) Sep 23 13:44:25.196494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 23 13:44:25.196494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 23 13:44:25.197657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 23 13:44:25.197657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 23 13:44:25.197657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 23 13:44:25.197657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 23 13:44:25.197657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 23 13:44:25.197657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 23 13:44:25.197657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 23 13:44:25.197657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 23 13:44:25.197657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 23 13:44:25.197657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 23 13:44:25.197657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 23 13:44:25.197657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 23 13:44:25.197657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 23 13:44:25.197657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 23 13:44:25.197657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 23 13:44:25.197657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception Sep 23 13:44:25.202763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 23 13:44:25.202763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 23 13:44:25.202763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 23 13:44:25.202763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 23 13:44:25.202763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 23 13:44:25.202763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application result = self._query(query) Sep 23 13:44:25.202763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 23 13:44:25.202763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application conn.query(q) Sep 23 13:44:25.202763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 23 13:44:25.202763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 23 13:44:25.202763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 23 13:44:25.202763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application result.read() Sep 23 13:44:25.202763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 23 13:44:25.202763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 23 13:44:25.202763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 23 13:44:25.202763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 23 13:44:25.203611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 23 13:44:25.203611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 23 13:44:25.203611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 23 13:44:25.203611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 23 13:44:25.203611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 23 13:44:25.203611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 23 13:44:25.203611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [parameters: {'user_id': '4c88fb5d45884fa68c7104c2b753f0ab', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 23 13:44:25.203611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 23 13:44:25.203611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:25.203611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:25.203611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:25.203611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:25.203611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:25.203611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:25.203611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:25.204418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:25.204418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:25.204418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:25.204418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:25.204418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:25.204418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:25.204418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:25.204418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 23 13:44:25.204418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 23 13:44:25.204418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:25.204418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:25.204418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:44:25.204418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:25.204418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:44:25.204418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:25.204418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 23 13:44:25.204418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 23 13:44:25.207314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 23 13:44:25.207314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 23 13:44:25.207314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 23 13:44:25.207314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 23 13:44:25.207314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 23 13:44:25.207314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:25.207314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 494/982] 10.222.0.62 () {60 vars in 1131 bytes} [Fri Sep 23 13:44:24 2022] POST /identity/v3/users => generated 160 bytes in 251 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:44:25.207314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2394a4f2-5868-4b90-bd68-abd9b66f8a24 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:25.217213 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:25.294221 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc90364-a381-4db5-a40f-af9d2c3cdc67 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:25.326464 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2394a4f2-5868-4b90-bd68-abd9b66f8a24 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:25.337015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83243e6d-cf3c-4ea3-9861-986c3de9f8ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:25.337941 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83243e6d-cf3c-4ea3-9861-986c3de9f8ce None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:25.338492 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83243e6d-cf3c-4ea3-9861-986c3de9f8ce None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:25.338865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83243e6d-cf3c-4ea3-9861-986c3de9f8ce None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:25.339663 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83243e6d-cf3c-4ea3-9861-986c3de9f8ce None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:25.340043 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83243e6d-cf3c-4ea3-9861-986c3de9f8ce None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:25.341523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83243e6d-cf3c-4ea3-9861-986c3de9f8ce None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:25.355491 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 489/983] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:44:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:25.356792 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a0d0ae-9a31-4745-a466-fa42242d028c None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:25.357080 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a0d0ae-9a31-4745-a466-fa42242d028c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:25.357353 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a0d0ae-9a31-4745-a466-fa42242d028c None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:25.363241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 490/984] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:25.363241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a0d0ae-9a31-4745-a466-fa42242d028c None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:25.372568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59ab582-8dff-4ce7-ada3-733f22a77837 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:25.373599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59ab582-8dff-4ce7-ada3-733f22a77837 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:25.373885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59ab582-8dff-4ce7-ada3-733f22a77837 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:25.374157 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59ab582-8dff-4ce7-ada3-733f22a77837 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:25.383182 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59ab582-8dff-4ce7-ada3-733f22a77837 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:25.384855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59ab582-8dff-4ce7-ada3-733f22a77837 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:25.396733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-e59ab582-8dff-4ce7-ada3-733f22a77837 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 23 13:44:25.396733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:25.396733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:25.396733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:25.396733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:25.396733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:25.396733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:25.396733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:25.396733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:25.396733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:25.396733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:25.396733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:25.396733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:44:25.396733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:44:25.396733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:44:25.396733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:44:25.397883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:25.397883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:25.397883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:44:25.397883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:25.397883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:44:25.397883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:25.397883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:25.397883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:25.397883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:25.397883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:25.397883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:25.397883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:25.397883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:25.397883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:25.397883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:25.397883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:25.397883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:25.397883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:25.399336 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:25.399336 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:25.399336 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:44:25.399336 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:44:25.399336 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:44:25.399336 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:44:25.399336 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:44:25.399336 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:44:25.399336 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 23 13:44:25.399336 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:25.402619 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 495/985] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:44:25 2022] GET /identity/v3/users/cinder => generated 84 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:25.403932 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a9ed73-3159-45c9-955f-9a8df5056b6e None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:25.404211 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a9ed73-3159-45c9-955f-9a8df5056b6e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:25.404481 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a9ed73-3159-45c9-955f-9a8df5056b6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:25.426708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-b2a9ed73-3159-45c9-955f-9a8df5056b6e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:25.428252 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-b2a9ed73-3159-45c9-955f-9a8df5056b6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:25.452383 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:25.475852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a0d0ae-9a31-4745-a466-fa42242d028c None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:25.507274 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:25.512481 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 496/986] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:25 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:25.513368 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a9ed73-3159-45c9-955f-9a8df5056b6e None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:25.524039 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0585a1b4-dba7-4490-ba51-849a973c8c84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:25.525184 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0585a1b4-dba7-4490-ba51-849a973c8c84 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:25.525906 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0585a1b4-dba7-4490-ba51-849a973c8c84 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:25.526869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0585a1b4-dba7-4490-ba51-849a973c8c84 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:25.527594 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0585a1b4-dba7-4490-ba51-849a973c8c84 None admin] RBAC: Adding query filter params (role.id=b2084c06d5a44297a9a7795d559fae94, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=b471d768cd814ff8b4f6bd35d21e24a8) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:25.528088 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0585a1b4-dba7-4490-ba51-849a973c8c84 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:25.530348 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0585a1b4-dba7-4490-ba51-849a973c8c84 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:25.546638 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 491/987] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:44:25 2022] GET /identity/v3/role_assignments?user.id=b471d768cd814ff8b4f6bd35d21e24a8&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=b2084c06d5a44297a9a7795d559fae94 => generated 608 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:25.633581 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a9ed73-3159-45c9-955f-9a8df5056b6e None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:25.671910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d297f8a-d18b-4aa0-98bd-c768cbe9b354 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:25.672535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d297f8a-d18b-4aa0-98bd-c768cbe9b354 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:25.672812 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d297f8a-d18b-4aa0-98bd-c768cbe9b354 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:25.673082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d297f8a-d18b-4aa0-98bd-c768cbe9b354 None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:25.674323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d297f8a-d18b-4aa0-98bd-c768cbe9b354 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:25.674323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d297f8a-d18b-4aa0-98bd-c768cbe9b354 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:25.676165 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d297f8a-d18b-4aa0-98bd-c768cbe9b354 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:25.718496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 497/988] 10.222.0.62 () {56 vars in 1101 bytes} [Fri Sep 23 13:44:25 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:25.787668 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0585a1b4-dba7-4490-ba51-849a973c8c84 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:25.909777 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0585a1b4-dba7-4490-ba51-849a973c8c84 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:25.944994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9172482e-9b8b-4498-8ed7-24355c8a53d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:25.945601 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9172482e-9b8b-4498-8ed7-24355c8a53d1 None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:25.945882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9172482e-9b8b-4498-8ed7-24355c8a53d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:25.946154 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9172482e-9b8b-4498-8ed7-24355c8a53d1 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:25.946840 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9172482e-9b8b-4498-8ed7-24355c8a53d1 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:25.948198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9172482e-9b8b-4498-8ed7-24355c8a53d1 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:26.037607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 492/989] 10.222.0.62 () {60 vars in 1141 bytes} [Fri Sep 23 13:44:25 2022] POST /identity/v3/services => generated 242 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:44:26.756071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d671570c-2816-4ba4-97ff-01e93e4e9011 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:26.756594 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d671570c-2816-4ba4-97ff-01e93e4e9011 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:26.756594 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d671570c-2816-4ba4-97ff-01e93e4e9011 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:26.758125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 498/990] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:26.763219 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4a40eb-3645-4faf-a6b3-198dc9da1c4c None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:26.763506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4a40eb-3645-4faf-a6b3-198dc9da1c4c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:26.763781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4a40eb-3645-4faf-a6b3-198dc9da1c4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:26.772219 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:26.782304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-ad4a40eb-3645-4faf-a6b3-198dc9da1c4c None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:26.784050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ad4a40eb-3645-4faf-a6b3-198dc9da1c4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:26.854419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 493/991] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:26 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:27.122660 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d671570c-2816-4ba4-97ff-01e93e4e9011 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:27.184443 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:27.237100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d671570c-2816-4ba4-97ff-01e93e4e9011 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:27.272641 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd43a8e9-efe2-4567-935a-2693867cceaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:27.273276 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd43a8e9-efe2-4567-935a-2693867cceaa None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:27.273616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd43a8e9-efe2-4567-935a-2693867cceaa None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:27.273893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd43a8e9-efe2-4567-935a-2693867cceaa None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:27.274510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd43a8e9-efe2-4567-935a-2693867cceaa None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:27.275880 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd43a8e9-efe2-4567-935a-2693867cceaa None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:27.285510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-dd43a8e9-efe2-4567-935a-2693867cceaa None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 23 13:44:27.285510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:27.285510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:44:27.285510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:44:27.285510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:44:27.285510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:44:27.285510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:44:27.285510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:27.285510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 23 13:44:27.285510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:27.285510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:27.285510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:27.285510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:27.285510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:27.285510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:27.285510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:27.285510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:27.286554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:27.286554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:27.286554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:27.286554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:27.286554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:27.286554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:27.286554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:44:27.286554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:44:27.286554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:44:27.286554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:44:27.286554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:27.286554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:27.286554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:27.286554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:27.286554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:27.286554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:27.287394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:27.287394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:27.287394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:27.287394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:27.287394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:27.287394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:27.287394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:27.287394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:27.287394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:27.287394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:27.287394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:44:27.287394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:27.287394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 23 13:44:27.287394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:27.287394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 499/992] 10.222.0.62 () {56 vars in 1126 bytes} [Fri Sep 23 13:44:27 2022] GET /identity/v3/services/object-store => generated 93 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:27.292444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4a40eb-3645-4faf-a6b3-198dc9da1c4c None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:27.393111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4a40eb-3645-4faf-a6b3-198dc9da1c4c None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:27.421437 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:27.427741 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf687f38-58cd-4278-bb77-f8eabf351955 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:27.437388 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf687f38-58cd-4278-bb77-f8eabf351955 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:27.437388 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf687f38-58cd-4278-bb77-f8eabf351955 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:27.437388 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf687f38-58cd-4278-bb77-f8eabf351955 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:27.437388 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf687f38-58cd-4278-bb77-f8eabf351955 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:27.437388 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf687f38-58cd-4278-bb77-f8eabf351955 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:27.437388 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf687f38-58cd-4278-bb77-f8eabf351955 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:27.437811 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 494/993] 10.222.0.62 () {56 vars in 1122 bytes} [Fri Sep 23 13:44:27 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:27.441291 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd43a8e9-efe2-4567-935a-2693867cceaa None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:27.541410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd43a8e9-efe2-4567-935a-2693867cceaa None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:27.564728 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:27.578530 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130a31f3-15d0-4e89-b756-77a5d9b91e79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:27.580113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130a31f3-15d0-4e89-b756-77a5d9b91e79 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:27.580534 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130a31f3-15d0-4e89-b756-77a5d9b91e79 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:27.581001 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130a31f3-15d0-4e89-b756-77a5d9b91e79 None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:27.582048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130a31f3-15d0-4e89-b756-77a5d9b91e79 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:27.582458 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130a31f3-15d0-4e89-b756-77a5d9b91e79 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:27.583834 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130a31f3-15d0-4e89-b756-77a5d9b91e79 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:27.589235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 500/994] 10.222.0.62 () {56 vars in 1122 bytes} [Fri Sep 23 13:44:27 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:27.595289 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf687f38-58cd-4278-bb77-f8eabf351955 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:27.698007 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf687f38-58cd-4278-bb77-f8eabf351955 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:27.731832 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cff4e47-569c-44ed-86aa-5bbade57f369 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:27.732438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cff4e47-569c-44ed-86aa-5bbade57f369 None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:27.732717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cff4e47-569c-44ed-86aa-5bbade57f369 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:27.732986 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cff4e47-569c-44ed-86aa-5bbade57f369 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:27.733658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cff4e47-569c-44ed-86aa-5bbade57f369 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:27.735037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cff4e47-569c-44ed-86aa-5bbade57f369 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:27.816723 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 495/995] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 13:44:27 2022] POST /identity/v3/endpoints => generated 325 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:44:27.946189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10428889-ac37-4fbb-a69f-e46a44283cbf None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:27.946744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10428889-ac37-4fbb-a69f-e46a44283cbf None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:27.946744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10428889-ac37-4fbb-a69f-e46a44283cbf None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:27.948400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 501/996] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:27.953823 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d779813-8a91-46ed-bdb1-b9df76215713 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:27.954111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d779813-8a91-46ed-bdb1-b9df76215713 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:27.954639 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d779813-8a91-46ed-bdb1-b9df76215713 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:27.974664 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-6d779813-8a91-46ed-bdb1-b9df76215713 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:27.975884 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-6d779813-8a91-46ed-bdb1-b9df76215713 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:28.028925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:28.053239 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 496/997] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:27 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:28.205943 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc800b00-edeb-46a6-9b32-753edc1048ea None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:28.206273 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc800b00-edeb-46a6-9b32-753edc1048ea None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:28.206520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc800b00-edeb-46a6-9b32-753edc1048ea None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:28.208012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 502/998] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:28.215009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b45c0c-c12f-4ae7-adbd-666b5258a4b7 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:28.215009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b45c0c-c12f-4ae7-adbd-666b5258a4b7 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:28.215151 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b45c0c-c12f-4ae7-adbd-666b5258a4b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:28.233714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-64b45c0c-c12f-4ae7-adbd-666b5258a4b7 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:28.234905 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-64b45c0c-c12f-4ae7-adbd-666b5258a4b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:28.306469 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 497/999] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:28 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:28.324683 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc800b00-edeb-46a6-9b32-753edc1048ea None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:28.426425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc800b00-edeb-46a6-9b32-753edc1048ea None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:28.460593 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472ed76f-c321-4531-b790-c7c6fc04eb49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:28.461193 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472ed76f-c321-4531-b790-c7c6fc04eb49 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:28.461470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472ed76f-c321-4531-b790-c7c6fc04eb49 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:28.461736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472ed76f-c321-4531-b790-c7c6fc04eb49 None admin] PATH_INFO: `/v3/services/image` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:28.462302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472ed76f-c321-4531-b790-c7c6fc04eb49 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:28.463662 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472ed76f-c321-4531-b790-c7c6fc04eb49 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:28.472685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-472ed76f-c321-4531-b790-c7c6fc04eb49 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 23 13:44:28.472685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:28.472685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:44:28.472685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:44:28.472685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:44:28.472685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:44:28.472685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:44:28.472685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:28.472685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 23 13:44:28.472685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:28.472685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:28.472685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:28.472685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:28.472685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:28.472685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:28.472685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:28.472685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:28.473679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:28.473679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:28.473679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:28.473679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:28.473679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:28.473679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:28.473679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:44:28.473679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:44:28.473679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:44:28.473679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:44:28.473679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:28.473679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:28.473679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:28.473679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:28.473679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:28.473679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:28.474564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:28.474564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:28.474564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:28.474564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:28.474564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:28.474564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:28.474564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:28.474564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:28.474564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:28.474564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:28.474564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:44:28.474564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:28.474564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 23 13:44:28.474564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:28.478763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 503/1000] 10.222.0.62 () {56 vars in 1105 bytes} [Fri Sep 23 13:44:28 2022] GET /identity/v3/services/image => generated 86 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:28.479543 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b45c0c-c12f-4ae7-adbd-666b5258a4b7 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:28.570804 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472ed76f-c321-4531-b790-c7c6fc04eb49 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:28.579250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b45c0c-c12f-4ae7-adbd-666b5258a4b7 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:28.590765 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:28.616226 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89307f01-e71b-49b3-9870-131901b5dea4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:28.616830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89307f01-e71b-49b3-9870-131901b5dea4 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:28.617112 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89307f01-e71b-49b3-9870-131901b5dea4 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:28.617388 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89307f01-e71b-49b3-9870-131901b5dea4 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:28.618075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89307f01-e71b-49b3-9870-131901b5dea4 None admin] RBAC: Adding query filter params (name=image) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:28.618403 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89307f01-e71b-49b3-9870-131901b5dea4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:28.619954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89307f01-e71b-49b3-9870-131901b5dea4 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:28.623878 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:28.629755 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 498/1001] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:28 2022] GET /identity/v3/services?name=image => generated 122 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:28.630891 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89307f01-e71b-49b3-9870-131901b5dea4 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:28.681201 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472ed76f-c321-4531-b790-c7c6fc04eb49 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:28.715838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d8d3ee-8b70-4ecf-bf20-dd07e09196a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:28.716386 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d8d3ee-8b70-4ecf-bf20-dd07e09196a1 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:28.716663 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d8d3ee-8b70-4ecf-bf20-dd07e09196a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:28.716933 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d8d3ee-8b70-4ecf-bf20-dd07e09196a1 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:28.717525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d8d3ee-8b70-4ecf-bf20-dd07e09196a1 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:28.718918 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d8d3ee-8b70-4ecf-bf20-dd07e09196a1 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:28.728760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-a3d8d3ee-8b70-4ecf-bf20-dd07e09196a1 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 23 13:44:28.728760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:28.728760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:44:28.728760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:44:28.728760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:44:28.728760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:44:28.728760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:44:28.728760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:28.728760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 23 13:44:28.728760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:28.728760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:28.728760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:28.728760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:28.728760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:28.728760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:28.728760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:28.728760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:28.729838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:28.729838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:28.729838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:28.729838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:28.729838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:28.729838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:28.729838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:44:28.729838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:44:28.729838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:44:28.729838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:44:28.729838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:28.729838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:28.729838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:28.729838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:28.729838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:28.729838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:28.731164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:28.731164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:28.731164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:28.731164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:28.731164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:28.731164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:28.731164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:28.731164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:28.731164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:28.731164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:28.731164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:44:28.731164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:28.731164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 23 13:44:28.731164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:28.731164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 504/1002] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:44:28 2022] GET /identity/v3/services/compute => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:28.737232 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d8d3ee-8b70-4ecf-bf20-dd07e09196a1 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:28.750251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89307f01-e71b-49b3-9870-131901b5dea4 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:28.788257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea88003-449c-401f-8202-f1905ab11986 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:28.788859 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea88003-449c-401f-8202-f1905ab11986 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:28.789145 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea88003-449c-401f-8202-f1905ab11986 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:28.789413 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea88003-449c-401f-8202-f1905ab11986 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:28.790027 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea88003-449c-401f-8202-f1905ab11986 None admin] RBAC: Adding query filter params (type=image) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:28.790350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea88003-449c-401f-8202-f1905ab11986 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:28.791708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea88003-449c-401f-8202-f1905ab11986 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:28.796443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 499/1003] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:28 2022] GET /identity/v3/services?type=image => generated 122 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:28.844498 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d8d3ee-8b70-4ecf-bf20-dd07e09196a1 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:28.861835 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:28.884441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e39f0c5-dc98-4003-a2f5-a0e056c6fac6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:28.885054 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e39f0c5-dc98-4003-a2f5-a0e056c6fac6 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:28.885329 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e39f0c5-dc98-4003-a2f5-a0e056c6fac6 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:28.885595 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e39f0c5-dc98-4003-a2f5-a0e056c6fac6 None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:28.886228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e39f0c5-dc98-4003-a2f5-a0e056c6fac6 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:28.886558 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e39f0c5-dc98-4003-a2f5-a0e056c6fac6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:28.887948 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e39f0c5-dc98-4003-a2f5-a0e056c6fac6 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:28.894090 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 505/1004] 10.222.0.62 () {56 vars in 1112 bytes} [Fri Sep 23 13:44:28 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:28.899673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea88003-449c-401f-8202-f1905ab11986 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:28.944407 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:29.006112 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea88003-449c-401f-8202-f1905ab11986 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:29.040082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8823f19c-9ddf-4379-ac6f-137a9b9136bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:29.040687 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8823f19c-9ddf-4379-ac6f-137a9b9136bd None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:29.040967 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8823f19c-9ddf-4379-ac6f-137a9b9136bd None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:29.041234 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8823f19c-9ddf-4379-ac6f-137a9b9136bd None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:29.041855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8823f19c-9ddf-4379-ac6f-137a9b9136bd None admin] RBAC: Adding query filter params (type=compute) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:29.042152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8823f19c-9ddf-4379-ac6f-137a9b9136bd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:29.043543 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8823f19c-9ddf-4379-ac6f-137a9b9136bd None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:29.048635 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 500/1005] 10.222.0.62 () {56 vars in 1112 bytes} [Fri Sep 23 13:44:28 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:29.054350 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e39f0c5-dc98-4003-a2f5-a0e056c6fac6 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:29.158601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e39f0c5-dc98-4003-a2f5-a0e056c6fac6 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:29.194083 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f594b7b-5abd-4438-a37f-3e18d5a78367 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:29.194716 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f594b7b-5abd-4438-a37f-3e18d5a78367 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:29.194993 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f594b7b-5abd-4438-a37f-3e18d5a78367 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:29.195260 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f594b7b-5abd-4438-a37f-3e18d5a78367 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:29.195954 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f594b7b-5abd-4438-a37f-3e18d5a78367 None admin] RBAC: Adding query filter params (interface=public, service_id=e21b149c49af4248896842203cdb8b05, region_id=RegionOne) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:29.196258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f594b7b-5abd-4438-a37f-3e18d5a78367 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:29.197681 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f594b7b-5abd-4438-a37f-3e18d5a78367 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:29.208466 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 506/1006] 10.222.0.62 () {56 vars in 1251 bytes} [Fri Sep 23 13:44:29 2022] GET /identity/v3/endpoints?service_id=e21b149c49af4248896842203cdb8b05&interface=public®ion_id=RegionOne => generated 194 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:29.215004 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8823f19c-9ddf-4379-ac6f-137a9b9136bd None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:29.315087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8823f19c-9ddf-4379-ac6f-137a9b9136bd None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:29.317312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:29.349672 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358f949e-1072-4e8b-a8dd-e54e1302ada8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:29.350273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358f949e-1072-4e8b-a8dd-e54e1302ada8 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:29.350576 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358f949e-1072-4e8b-a8dd-e54e1302ada8 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:29.350841 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358f949e-1072-4e8b-a8dd-e54e1302ada8 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:29.351436 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358f949e-1072-4e8b-a8dd-e54e1302ada8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:29.352774 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358f949e-1072-4e8b-a8dd-e54e1302ada8 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:29.359014 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 501/1007] 10.222.0.62 () {56 vars in 1087 bytes} [Fri Sep 23 13:44:29 2022] GET /identity/v3/services => generated 1727 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 13:44:30.058060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899f6e83-21f5-4f7b-9361-1e59e7788175 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:30.058603 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899f6e83-21f5-4f7b-9361-1e59e7788175 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:30.058682 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899f6e83-21f5-4f7b-9361-1e59e7788175 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:30.060123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 507/1008] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:30.065932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8750c700-a0e4-4c95-9775-a9ccad030712 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:30.066216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8750c700-a0e4-4c95-9775-a9ccad030712 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:30.066526 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8750c700-a0e4-4c95-9775-a9ccad030712 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:30.084945 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-8750c700-a0e4-4c95-9775-a9ccad030712 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:30.086136 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8750c700-a0e4-4c95-9775-a9ccad030712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:30.154352 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 502/1009] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:30 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:30.428484 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899f6e83-21f5-4f7b-9361-1e59e7788175 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:30.474249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:30.532969 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899f6e83-21f5-4f7b-9361-1e59e7788175 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:30.567456 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3107b157-d231-448e-bfe3-4772175b2a9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:30.568092 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3107b157-d231-448e-bfe3-4772175b2a9b None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:30.568373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3107b157-d231-448e-bfe3-4772175b2a9b None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:30.568643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3107b157-d231-448e-bfe3-4772175b2a9b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:30.569778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3107b157-d231-448e-bfe3-4772175b2a9b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:30.571242 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3107b157-d231-448e-bfe3-4772175b2a9b None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:30.572986 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 508/1010] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:30.578730 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8750c700-a0e4-4c95-9775-a9ccad030712 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:30.672571 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:30.681552 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8750c700-a0e4-4c95-9775-a9ccad030712 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:30.719435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38695cc1-e8b2-412d-b17f-f87fbc38346b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:30.720607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38695cc1-e8b2-412d-b17f-f87fbc38346b None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:30.720891 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38695cc1-e8b2-412d-b17f-f87fbc38346b None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:30.721115 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38695cc1-e8b2-412d-b17f-f87fbc38346b None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:30.721774 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38695cc1-e8b2-412d-b17f-f87fbc38346b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:30.723219 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38695cc1-e8b2-412d-b17f-f87fbc38346b None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:30.818078 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 503/1011] 10.222.0.62 () {60 vars in 1141 bytes} [Fri Sep 23 13:44:30 2022] POST /identity/v3/projects => generated 311 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:44:30.961132 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc0a65b-1b7b-4ea7-bb49-0bf026784d8c None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:30.961493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc0a65b-1b7b-4ea7-bb49-0bf026784d8c None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:30.961659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc0a65b-1b7b-4ea7-bb49-0bf026784d8c None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:30.963258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 509/1012] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:30.968763 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66956d4-5478-4b8b-9d55-302f17a15001 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:30.969064 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66956d4-5478-4b8b-9d55-302f17a15001 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:30.969316 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66956d4-5478-4b8b-9d55-302f17a15001 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:30.988679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-e66956d4-5478-4b8b-9d55-302f17a15001 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:30.990134 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e66956d4-5478-4b8b-9d55-302f17a15001 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:31.063925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 504/1013] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:30 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:31.334373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc0a65b-1b7b-4ea7-bb49-0bf026784d8c None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:31.447142 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc0a65b-1b7b-4ea7-bb49-0bf026784d8c None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:31.458980 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:31.482660 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e47f0b-9834-487f-aa32-8f8ca454f4f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:31.483193 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e47f0b-9834-487f-aa32-8f8ca454f4f6 None admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:31.483495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e47f0b-9834-487f-aa32-8f8ca454f4f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:31.483767 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e47f0b-9834-487f-aa32-8f8ca454f4f6 None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:31.484424 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e47f0b-9834-487f-aa32-8f8ca454f4f6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:31.485974 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e47f0b-9834-487f-aa32-8f8ca454f4f6 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:31.499893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39645826-07dd-4716-a63c-818d52673bd3 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:31.500182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39645826-07dd-4716-a63c-818d52673bd3 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:31.500478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39645826-07dd-4716-a63c-818d52673bd3 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:31.502570 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 505/1014] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:44:31.507125 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5771e9a2-f770-4876-b10f-ece33fadd247 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:31.507428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5771e9a2-f770-4876-b10f-ece33fadd247 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:31.507696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5771e9a2-f770-4876-b10f-ece33fadd247 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:31.526127 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-5771e9a2-f770-4876-b10f-ece33fadd247 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:31.527336 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-5771e9a2-f770-4876-b10f-ece33fadd247 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:31.586231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 510/1015] 10.222.0.62 () {60 vars in 1141 bytes} [Fri Sep 23 13:44:31 2022] POST /identity/v3/services => generated 242 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:44:31.607381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 506/1016] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:31 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 23 13:44:31.875295 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e47f0b-9834-487f-aa32-8f8ca454f4f6 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:31.979327 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e47f0b-9834-487f-aa32-8f8ca454f4f6 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:32.014426 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6ff983-e78e-4414-8e71-b07c8f193b8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:32.015430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6ff983-e78e-4414-8e71-b07c8f193b8c None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:32.015940 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6ff983-e78e-4414-8e71-b07c8f193b8c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:32.017874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6ff983-e78e-4414-8e71-b07c8f193b8c None admin] PATH_INFO: `/v3/services/compute` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:32.018707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6ff983-e78e-4414-8e71-b07c8f193b8c None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:32.020272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6ff983-e78e-4414-8e71-b07c8f193b8c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:32.029409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-fd6ff983-e78e-4414-8e71-b07c8f193b8c None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 23 13:44:32.029409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:32.029409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:44:32.029409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:44:32.029409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:44:32.029409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:44:32.029409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:44:32.029409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:32.029409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 23 13:44:32.029409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:32.029409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:32.029409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:32.029409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:32.029409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:32.029409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:32.029409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:32.029409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:32.030823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:32.030823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:32.030823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:32.030823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:32.030823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:32.030823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:32.030823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:44:32.030823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:44:32.030823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:44:32.030823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:44:32.030823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:32.030823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:32.030823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:32.030823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:32.030823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:32.030823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:32.031677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:32.031677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:32.031677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:32.031677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:32.031677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:32.031677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:32.031677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:32.031677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:32.031677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:32.031677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:32.031677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:44:32.031677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:32.031677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 23 13:44:32.031677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:32.031677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 511/1017] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:44:31 2022] GET /identity/v3/services/compute => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:32.037717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5771e9a2-f770-4876-b10f-ece33fadd247 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:32.041896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:32.141212 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5771e9a2-f770-4876-b10f-ece33fadd247 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:32.175555 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839281b1-1f50-4134-99de-4526b92bdf48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:32.176160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839281b1-1f50-4134-99de-4526b92bdf48 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:32.176433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839281b1-1f50-4134-99de-4526b92bdf48 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:32.176701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-839281b1-1f50-4134-99de-4526b92bdf48 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:32.177322 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839281b1-1f50-4134-99de-4526b92bdf48 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:32.177616 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839281b1-1f50-4134-99de-4526b92bdf48 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:32.179053 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-839281b1-1f50-4134-99de-4526b92bdf48 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:32.184068 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 507/1018] 10.222.0.62 () {56 vars in 1112 bytes} [Fri Sep 23 13:44:32 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:32.190142 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6ff983-e78e-4414-8e71-b07c8f193b8c None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:32.228155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:32.294436 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6ff983-e78e-4414-8e71-b07c8f193b8c None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:32.328752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f770be2e-0b9b-4192-83ad-1cbb2a6de8dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:32.329946 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f770be2e-0b9b-4192-83ad-1cbb2a6de8dd None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:32.330154 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f770be2e-0b9b-4192-83ad-1cbb2a6de8dd None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:32.330494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f770be2e-0b9b-4192-83ad-1cbb2a6de8dd None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:32.331117 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f770be2e-0b9b-4192-83ad-1cbb2a6de8dd None admin] RBAC: Adding query filter params (type=compute) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:32.331413 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f770be2e-0b9b-4192-83ad-1cbb2a6de8dd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:32.332760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f770be2e-0b9b-4192-83ad-1cbb2a6de8dd None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:32.337846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 512/1019] 10.222.0.62 () {56 vars in 1112 bytes} [Fri Sep 23 13:44:32 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:32.343291 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839281b1-1f50-4134-99de-4526b92bdf48 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:32.409425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:32.442880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-839281b1-1f50-4134-99de-4526b92bdf48 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:32.476668 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f50b18-81f5-483c-ba59-6ade33e40592 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:32.477266 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f50b18-81f5-483c-ba59-6ade33e40592 None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:32.477540 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f50b18-81f5-483c-ba59-6ade33e40592 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:32.477809 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f50b18-81f5-483c-ba59-6ade33e40592 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:32.478501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f50b18-81f5-483c-ba59-6ade33e40592 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:32.479850 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f50b18-81f5-483c-ba59-6ade33e40592 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:32.556736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 508/1020] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 13:44:32 2022] POST /identity/v3/endpoints => generated 334 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:44:33.038673 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85412a6d-1d92-4c51-aba0-53566ceb5548 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:33.038999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85412a6d-1d92-4c51-aba0-53566ceb5548 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:33.039222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85412a6d-1d92-4c51-aba0-53566ceb5548 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:33.040602 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 513/1021] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:33.045631 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a7bf19-ab74-4451-8100-4d7eb9aeeff8 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:33.046058 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a7bf19-ab74-4451-8100-4d7eb9aeeff8 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:33.046166 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a7bf19-ab74-4451-8100-4d7eb9aeeff8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:33.064930 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-62a7bf19-ab74-4451-8100-4d7eb9aeeff8 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:33.066123 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-62a7bf19-ab74-4451-8100-4d7eb9aeeff8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:33.141830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 509/1022] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:33 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:33.410771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85412a6d-1d92-4c51-aba0-53566ceb5548 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:33.510058 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:33.515547 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85412a6d-1d92-4c51-aba0-53566ceb5548 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:33.552691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea9aead-8af7-4766-86f1-4c0ad7d24fd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:33.553326 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea9aead-8af7-4766-86f1-4c0ad7d24fd2 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:33.553606 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea9aead-8af7-4766-86f1-4c0ad7d24fd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:33.553878 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea9aead-8af7-4766-86f1-4c0ad7d24fd2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:33.555034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea9aead-8af7-4766-86f1-4c0ad7d24fd2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:33.556577 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea9aead-8af7-4766-86f1-4c0ad7d24fd2 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:33.558363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 514/1023] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:33 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:33.564392 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a7bf19-ab74-4451-8100-4d7eb9aeeff8 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:33.614471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:33.667063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a7bf19-ab74-4451-8100-4d7eb9aeeff8 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:33.704794 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b117df-1094-4af5-9e7d-868d55c20b1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:33.705409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b117df-1094-4af5-9e7d-868d55c20b1f None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:33.705691 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b117df-1094-4af5-9e7d-868d55c20b1f None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:33.705961 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b117df-1094-4af5-9e7d-868d55c20b1f None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:33.706647 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b117df-1094-4af5-9e7d-868d55c20b1f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:33.708068 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b117df-1094-4af5-9e7d-868d55c20b1f None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:33.775363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04df3c5e-15a8-444f-9c27-6f2412dda6c3 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:33.775653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04df3c5e-15a8-444f-9c27-6f2412dda6c3 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:33.775928 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04df3c5e-15a8-444f-9c27-6f2412dda6c3 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:33.777362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 515/1024] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:44:33.782598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea35292-140c-40dc-b7c5-0fa06765f9bd None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:33.782882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea35292-140c-40dc-b7c5-0fa06765f9bd None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:33.783151 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea35292-140c-40dc-b7c5-0fa06765f9bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:33.801188 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-3ea35292-140c-40dc-b7c5-0fa06765f9bd None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:33.802391 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-3ea35292-140c-40dc-b7c5-0fa06765f9bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:33.834475 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 510/1025] 10.222.0.62 () {60 vars in 1132 bytes} [Fri Sep 23 13:44:33 2022] POST /identity/v3/users => generated 285 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:44:33.889617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 516/1026] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:33 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 23 13:44:34.163895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b117df-1094-4af5-9e7d-868d55c20b1f None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:34.273070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b117df-1094-4af5-9e7d-868d55c20b1f None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:34.280343 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:34.307883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50d0d14-e2ae-49e1-8f47-c669e0c69808 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:34.308590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50d0d14-e2ae-49e1-8f47-c669e0c69808 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:34.308906 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50d0d14-e2ae-49e1-8f47-c669e0c69808 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:34.309177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50d0d14-e2ae-49e1-8f47-c669e0c69808 None admin] PATH_INFO: `/v3/services/image` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:34.309759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50d0d14-e2ae-49e1-8f47-c669e0c69808 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:34.311145 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50d0d14-e2ae-49e1-8f47-c669e0c69808 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:34.321630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-c50d0d14-e2ae-49e1-8f47-c669e0c69808 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 23 13:44:34.321630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:34.321630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:44:34.321630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:44:34.321630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:44:34.321630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:44:34.321630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:44:34.321630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:34.321630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 23 13:44:34.321630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:34.321630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:34.321630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:34.321630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:34.321630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:34.321630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:34.321630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:34.321630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:34.323896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:34.323896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:34.323896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:34.323896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:34.323896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:34.323896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:34.323896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:44:34.323896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:44:34.323896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:44:34.323896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:44:34.323896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:34.323896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:34.323896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:34.323896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:34.323896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:34.323896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:34.324756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:34.324756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:34.324756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:34.324756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:34.324756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:34.324756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:34.324756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:34.324756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:34.324756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:34.324756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:34.324756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:44:34.324756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:34.324756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 23 13:44:34.324756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:34.324756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 511/1027] 10.222.0.62 () {56 vars in 1105 bytes} [Fri Sep 23 13:44:34 2022] GET /identity/v3/services/image => generated 86 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:34.329474 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea35292-140c-40dc-b7c5-0fa06765f9bd None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:34.427520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:34.434043 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea35292-140c-40dc-b7c5-0fa06765f9bd None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:34.468705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40e48dc-cd7b-48c5-863f-4eea2903ae85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:34.469950 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40e48dc-cd7b-48c5-863f-4eea2903ae85 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:34.470228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40e48dc-cd7b-48c5-863f-4eea2903ae85 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:34.470515 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40e48dc-cd7b-48c5-863f-4eea2903ae85 None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:34.471138 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40e48dc-cd7b-48c5-863f-4eea2903ae85 None admin] RBAC: Adding query filter params (name=image) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:34.471434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40e48dc-cd7b-48c5-863f-4eea2903ae85 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:34.472780 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40e48dc-cd7b-48c5-863f-4eea2903ae85 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:34.482172 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 517/1028] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:34 2022] GET /identity/v3/services?name=image => generated 122 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:34.483451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50d0d14-e2ae-49e1-8f47-c669e0c69808 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:34.585902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50d0d14-e2ae-49e1-8f47-c669e0c69808 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:34.621215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581f5056-638a-48a4-8de5-9860da088f49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:34.621829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581f5056-638a-48a4-8de5-9860da088f49 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:34.622111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581f5056-638a-48a4-8de5-9860da088f49 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:34.622406 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581f5056-638a-48a4-8de5-9860da088f49 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:34.623041 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581f5056-638a-48a4-8de5-9860da088f49 None admin] RBAC: Adding query filter params (type=image) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:34.623346 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581f5056-638a-48a4-8de5-9860da088f49 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:34.624713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581f5056-638a-48a4-8de5-9860da088f49 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:34.630049 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 512/1029] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:34 2022] GET /identity/v3/services?type=image => generated 350 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:34.635696 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40e48dc-cd7b-48c5-863f-4eea2903ae85 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:34.743567 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40e48dc-cd7b-48c5-863f-4eea2903ae85 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:34.778533 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb47ac72-85dd-4c71-bb48-9b9f5eb9a1a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:34.779144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47ac72-85dd-4c71-bb48-9b9f5eb9a1a0 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:34.779421 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47ac72-85dd-4c71-bb48-9b9f5eb9a1a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:34.779688 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47ac72-85dd-4c71-bb48-9b9f5eb9a1a0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:34.780454 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb47ac72-85dd-4c71-bb48-9b9f5eb9a1a0 None admin] RBAC: Adding query filter params (interface=public, service_id=416ffbfa8b844fffa87a8d6e65e34c9f, region_id=RegionOne) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:34.780762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb47ac72-85dd-4c71-bb48-9b9f5eb9a1a0 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:34.782746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb47ac72-85dd-4c71-bb48-9b9f5eb9a1a0 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:34.793475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 518/1030] 10.222.0.62 () {56 vars in 1251 bytes} [Fri Sep 23 13:44:34 2022] GET /identity/v3/endpoints?service_id=416ffbfa8b844fffa87a8d6e65e34c9f&interface=public®ion_id=RegionOne => generated 194 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:34.795283 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581f5056-638a-48a4-8de5-9860da088f49 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:34.814998 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d06fed-ad1d-4f1c-a64f-583865daded0 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:34.815277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d06fed-ad1d-4f1c-a64f-583865daded0 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:34.815550 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d06fed-ad1d-4f1c-a64f-583865daded0 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:34.817020 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 519/1031] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:44:34.821695 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec72f8e-71ff-4c76-b39a-22ccf35f1b96 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:34.821973 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec72f8e-71ff-4c76-b39a-22ccf35f1b96 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:34.822238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec72f8e-71ff-4c76-b39a-22ccf35f1b96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:34.831851 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:34.841136 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-0ec72f8e-71ff-4c76-b39a-22ccf35f1b96 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:34.842351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-0ec72f8e-71ff-4c76-b39a-22ccf35f1b96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:34.862674 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:34.901003 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581f5056-638a-48a4-8de5-9860da088f49 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:34.914320 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 520/1032] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:34 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 23 13:44:34.934920 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889a1999-fdbc-43b0-a6cd-64c21c387b2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:34.935525 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889a1999-fdbc-43b0-a6cd-64c21c387b2a None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:34.935798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889a1999-fdbc-43b0-a6cd-64c21c387b2a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:34.936060 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889a1999-fdbc-43b0-a6cd-64c21c387b2a None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:34.936649 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889a1999-fdbc-43b0-a6cd-64c21c387b2a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:34.938139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889a1999-fdbc-43b0-a6cd-64c21c387b2a None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:34.944769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 513/1033] 10.222.0.62 () {56 vars in 1087 bytes} [Fri Sep 23 13:44:34 2022] GET /identity/v3/services => generated 1957 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 13:44:35.193502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec72f8e-71ff-4c76-b39a-22ccf35f1b96 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:35.297764 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec72f8e-71ff-4c76-b39a-22ccf35f1b96 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:35.332271 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee299f6-761d-409e-b9e0-9df9424e071b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:35.332899 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee299f6-761d-409e-b9e0-9df9424e071b None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:35.333174 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee299f6-761d-409e-b9e0-9df9424e071b None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:35.333442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee299f6-761d-409e-b9e0-9df9424e071b None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:35.343421 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee299f6-761d-409e-b9e0-9df9424e071b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:44:35.348988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-4ee299f6-761d-409e-b9e0-9df9424e071b None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 23 13:44:35.348988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:35.348988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:35.348988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:35.348988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:35.348988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:35.348988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:35.348988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:35.348988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:35.348988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:35.348988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:35.348988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:35.348988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:44:35.348988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:44:35.348988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:44:35.348988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:44:35.349963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:44:35.349963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:44:35.349963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:44:35.349963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:44:35.349963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:35.349963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:35.349963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:35.349963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:35.349963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:35.349963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:35.349963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:35.349963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:35.349963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:35.349963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:35.349963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:35.349963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:35.351129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:35.351129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:35.351129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:35.351129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:35.351129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:44:35.351129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:44:35.351129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:44:35.351129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:44:35.351129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:44:35.351129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:44:35.351129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 23 13:44:35.351129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:35.352129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 521/1034] 10.222.0.62 () {56 vars in 1120 bytes} [Fri Sep 23 13:44:35 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:35.356314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889a1999-fdbc-43b0-a6cd-64c21c387b2a None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:35.456654 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:35.460956 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889a1999-fdbc-43b0-a6cd-64c21c387b2a None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:35.494995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8439ad93-03fe-443e-b7fd-eb65113078c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:35.495598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8439ad93-03fe-443e-b7fd-eb65113078c1 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:35.495916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8439ad93-03fe-443e-b7fd-eb65113078c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:35.496195 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8439ad93-03fe-443e-b7fd-eb65113078c1 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:35.496848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8439ad93-03fe-443e-b7fd-eb65113078c1 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:35.497145 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8439ad93-03fe-443e-b7fd-eb65113078c1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:35.498580 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8439ad93-03fe-443e-b7fd-eb65113078c1 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:35.507200 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 514/1035] 10.222.0.62 () {56 vars in 1115 bytes} [Fri Sep 23 13:44:35 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:35.513067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee299f6-761d-409e-b9e0-9df9424e071b None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:35.529973 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:35.614167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee299f6-761d-409e-b9e0-9df9424e071b None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:35.648907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c02055-d0ff-4bc8-9a17-08e39f28d7ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:35.649541 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c02055-d0ff-4bc8-9a17-08e39f28d7ff None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:35.649817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c02055-d0ff-4bc8-9a17-08e39f28d7ff None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:35.650082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c02055-d0ff-4bc8-9a17-08e39f28d7ff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:35.660997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c02055-d0ff-4bc8-9a17-08e39f28d7ff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:35.662477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c02055-d0ff-4bc8-9a17-08e39f28d7ff None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:35.677249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-92c02055-d0ff-4bc8-9a17-08e39f28d7ff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:35.677249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:35.677249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:35.677249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:35.677249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:35.677249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:35.677249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:35.677249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:35.677249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:35.677249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:35.677249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:35.677249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:35.677249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:35.677249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:35.677249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:35.677249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:35.678251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:35.678251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:35.678251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:35.678251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:35.678251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:35.678251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:35.678251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:35.678251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:35.678251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:35.678251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:35.678251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:35.678251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:35.678251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:35.678251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:35.678251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:35.678251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:35.678251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:35.679207 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:35.679207 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:35.679207 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:35.679207 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 522/1036] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:35.685130 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8439ad93-03fe-443e-b7fd-eb65113078c1 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:35.786059 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8439ad93-03fe-443e-b7fd-eb65113078c1 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:35.820067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2655a24c-47aa-47e9-8076-121a68b4b0c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:35.820702 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2655a24c-47aa-47e9-8076-121a68b4b0c4 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:35.820977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2655a24c-47aa-47e9-8076-121a68b4b0c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:35.821242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2655a24c-47aa-47e9-8076-121a68b4b0c4 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:35.821975 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2655a24c-47aa-47e9-8076-121a68b4b0c4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:35.822277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2655a24c-47aa-47e9-8076-121a68b4b0c4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:35.823688 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2655a24c-47aa-47e9-8076-121a68b4b0c4 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:35.841313 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 515/1037] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:35.842761 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c02055-d0ff-4bc8-9a17-08e39f28d7ff None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:35.911256 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:35.947079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c02055-d0ff-4bc8-9a17-08e39f28d7ff None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:35.982993 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1e0877-028d-4348-b686-7dba89b3240a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:35.983649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1e0877-028d-4348-b686-7dba89b3240a None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:35.983928 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1e0877-028d-4348-b686-7dba89b3240a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:35.984197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1e0877-028d-4348-b686-7dba89b3240a None admin] PATH_INFO: `/v3/users/nova` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:35.994274 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1e0877-028d-4348-b686-7dba89b3240a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:35.995762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1e0877-028d-4348-b686-7dba89b3240a None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:36.008941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-5d1e0877-028d-4348-b686-7dba89b3240a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 23 13:44:36.008941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:36.008941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:36.008941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:36.008941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:36.008941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:36.008941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:36.008941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:36.008941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:36.008941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:36.008941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:36.008941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:36.008941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:44:36.008941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:44:36.008941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:44:36.008941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:44:36.009961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:36.009961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:36.009961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:44:36.009961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:36.009961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:44:36.009961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:36.009961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:36.009961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:36.009961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:36.009961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:36.009961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:36.009961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:36.009961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:36.009961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:36.009961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:36.009961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:36.009961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:36.009961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:36.011318 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:36.011318 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:36.011318 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:44:36.011318 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:44:36.011318 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:44:36.011318 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:44:36.011318 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:44:36.011318 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:44:36.011318 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 23 13:44:36.011318 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:36.014401 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 523/1038] 10.222.0.62 () {56 vars in 1093 bytes} [Fri Sep 23 13:44:35 2022] GET /identity/v3/users/nova => generated 82 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:36.015980 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2655a24c-47aa-47e9-8076-121a68b4b0c4 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:36.039848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:36.091659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aac9dea-0207-48ca-b6c4-a407857bbc8f None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:36.091969 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aac9dea-0207-48ca-b6c4-a407857bbc8f None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:36.092209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aac9dea-0207-48ca-b6c4-a407857bbc8f None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:36.093687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 524/1039] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:44:36.098680 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c3b31f-7b62-49e6-a3d2-a20ab975aa1d None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:36.098958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c3b31f-7b62-49e6-a3d2-a20ab975aa1d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:36.099222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c3b31f-7b62-49e6-a3d2-a20ab975aa1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:36.118884 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-73c3b31f-7b62-49e6-a3d2-a20ab975aa1d None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:36.120048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-73c3b31f-7b62-49e6-a3d2-a20ab975aa1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:36.121300 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2655a24c-47aa-47e9-8076-121a68b4b0c4 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:36.163879 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5774aa6f-af6b-49f7-9a5a-04a91ccd5019 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:36.164554 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774aa6f-af6b-49f7-9a5a-04a91ccd5019 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:36.164834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774aa6f-af6b-49f7-9a5a-04a91ccd5019 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:36.165102 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774aa6f-af6b-49f7-9a5a-04a91ccd5019 None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:36.165834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5774aa6f-af6b-49f7-9a5a-04a91ccd5019 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:36.166130 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5774aa6f-af6b-49f7-9a5a-04a91ccd5019 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:36.167568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5774aa6f-af6b-49f7-9a5a-04a91ccd5019 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:36.193630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 516/1040] 10.222.0.62 () {56 vars in 1133 bytes} [Fri Sep 23 13:44:36 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:36.194685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5774aa6f-af6b-49f7-9a5a-04a91ccd5019 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:36.196402 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 525/1041] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:36 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 23 13:44:36.276894 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:36.298038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5774aa6f-af6b-49f7-9a5a-04a91ccd5019 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:36.333387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98272a7e-9503-4c24-a708-8166d8964813 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:36.333989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98272a7e-9503-4c24-a708-8166d8964813 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:36.334266 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98272a7e-9503-4c24-a708-8166d8964813 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:36.334574 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98272a7e-9503-4c24-a708-8166d8964813 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:36.400660 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98272a7e-9503-4c24-a708-8166d8964813 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:36.402082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98272a7e-9503-4c24-a708-8166d8964813 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:36.465426 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c3b31f-7b62-49e6-a3d2-a20ab975aa1d None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:36.479780 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:36.502840 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 517/1042] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:44:36 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 23 13:44:36.508312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98272a7e-9503-4c24-a708-8166d8964813 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:36.523337 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:36.570120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c3b31f-7b62-49e6-a3d2-a20ab975aa1d None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:36.604713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e1681f-0271-465d-90ea-98277b4d44d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:36.605344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e1681f-0271-465d-90ea-98277b4d44d8 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:36.605614 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e1681f-0271-465d-90ea-98277b4d44d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:36.605883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e1681f-0271-465d-90ea-98277b4d44d8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:36.617355 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98272a7e-9503-4c24-a708-8166d8964813 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:36.617513 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e1681f-0271-465d-90ea-98277b4d44d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:44:36.622131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-35e1681f-0271-465d-90ea-98277b4d44d8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:44:36.622131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:36.622131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:36.622131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:36.622131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:36.622131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:36.622131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:36.622131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:36.622131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:36.622131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:36.622131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:36.622131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:36.622131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:44:36.622131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:44:36.622131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:44:36.622131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:44:36.623181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:44:36.623181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:44:36.623181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:44:36.623181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:44:36.623181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:36.623181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:36.623181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:36.623181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:36.623181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:36.623181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:36.623181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:36.623181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:36.623181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:36.623181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:36.623181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:36.623181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:36.624643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:36.624643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:36.624643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:36.624643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:36.624643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:44:36.624643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:44:36.624643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:44:36.624643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:44:36.624643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:44:36.624643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:44:36.624643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:44:36.624643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:36.624643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 526/1043] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:44:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:44:36.629836 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e1681f-0271-465d-90ea-98277b4d44d8 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:36.656676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31fbabb9-5237-42d9-92ac-e31321654085 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:36.657414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fbabb9-5237-42d9-92ac-e31321654085 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:36.657690 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fbabb9-5237-42d9-92ac-e31321654085 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:36.657958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fbabb9-5237-42d9-92ac-e31321654085 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:36.669076 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31fbabb9-5237-42d9-92ac-e31321654085 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:36.670538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31fbabb9-5237-42d9-92ac-e31321654085 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:36.686456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-31fbabb9-5237-42d9-92ac-e31321654085 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:36.686456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:36.686456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:36.686456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:36.686456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:36.686456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:36.686456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:36.686456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:36.686456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:36.686456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:36.686456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:36.686456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:36.686456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:36.686456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:36.686456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:36.686456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:36.687421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:36.687421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:36.687421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:36.687421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:36.687421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:36.687421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:36.687421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:36.687421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:36.687421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:36.687421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:36.687421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:36.687421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:36.687421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:36.687421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:36.687421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:36.687421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:36.687421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:36.688895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:36.688895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:36.688895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:36.688895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 518/1044] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:44:36.694098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31fbabb9-5237-42d9-92ac-e31321654085 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:36.724618 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:36.738980 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e1681f-0271-465d-90ea-98277b4d44d8 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:36.757213 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:36.775483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f0d6a2-8575-451a-b8ad-8445a4e88e7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:36.776090 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f0d6a2-8575-451a-b8ad-8445a4e88e7f None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:36.776369 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f0d6a2-8575-451a-b8ad-8445a4e88e7f None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:36.776641 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92f0d6a2-8575-451a-b8ad-8445a4e88e7f None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:36.777306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f0d6a2-8575-451a-b8ad-8445a4e88e7f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:36.777607 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f0d6a2-8575-451a-b8ad-8445a4e88e7f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:36.778997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92f0d6a2-8575-451a-b8ad-8445a4e88e7f None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:36.787856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 527/1045] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:44:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:36.793923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f0d6a2-8575-451a-b8ad-8445a4e88e7f None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:36.799643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31fbabb9-5237-42d9-92ac-e31321654085 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:36.838994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee473cf9-e53b-47d2-9565-122d886ce9c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:36.839521 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee473cf9-e53b-47d2-9565-122d886ce9c9 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:36.839800 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee473cf9-e53b-47d2-9565-122d886ce9c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:36.840187 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee473cf9-e53b-47d2-9565-122d886ce9c9 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:36.840823 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee473cf9-e53b-47d2-9565-122d886ce9c9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:36.841119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee473cf9-e53b-47d2-9565-122d886ce9c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:36.842708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee473cf9-e53b-47d2-9565-122d886ce9c9 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:36.855990 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 519/1046] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:36.861977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee473cf9-e53b-47d2-9565-122d886ce9c9 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:36.897425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92f0d6a2-8575-451a-b8ad-8445a4e88e7f None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:36.931043 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7185ee0b-a086-4e10-9a3a-73afa94425d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:36.931714 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7185ee0b-a086-4e10-9a3a-73afa94425d2 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:36.931995 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7185ee0b-a086-4e10-9a3a-73afa94425d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:36.932261 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7185ee0b-a086-4e10-9a3a-73afa94425d2 None admin] PATH_INFO: `/v3/users/68fae1e50b4347b692dc718993d6fe73` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:36.947421 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7185ee0b-a086-4e10-9a3a-73afa94425d2 None admin] RBAC: Authorizing `identity:get_user(user_id=68fae1e50b4347b692dc718993d6fe73)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:36.948857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7185ee0b-a086-4e10-9a3a-73afa94425d2 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:36.950688 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 528/1047] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:44:36 2022] GET /identity/v3/users/68fae1e50b4347b692dc718993d6fe73 => generated 285 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:36.956127 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7185ee0b-a086-4e10-9a3a-73afa94425d2 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:36.957744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:36.969201 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee473cf9-e53b-47d2-9565-122d886ce9c9 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:36.993486 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:37.006797 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e985e2-ffcb-4b27-b4c2-84d6a579ff29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:37.007442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e985e2-ffcb-4b27-b4c2-84d6a579ff29 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:37.007747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e985e2-ffcb-4b27-b4c2-84d6a579ff29 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:37.007998 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e985e2-ffcb-4b27-b4c2-84d6a579ff29 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:37.016700 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e985e2-ffcb-4b27-b4c2-84d6a579ff29 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:37.017974 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e985e2-ffcb-4b27-b4c2-84d6a579ff29 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:37.031072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-d6e985e2-ffcb-4b27-b4c2-84d6a579ff29 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:37.031072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:37.031072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:37.031072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:37.031072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:37.031072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:37.031072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:37.031072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:37.031072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:37.031072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:37.031072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:37.031072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:37.031072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:44:37.031072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:44:37.031072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:44:37.032669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:44:37.032669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:37.032669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:37.032669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:37.032669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:37.032669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:37.032669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:37.032669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:37.032669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:37.032669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:37.032669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:37.032669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:37.032669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:37.032669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:37.032669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:37.032669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:37.032669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:37.034177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:44:37.034177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:44:37.034177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:44:37.034177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:44:37.034177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:44:37.034177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:44:37.034177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:37.034177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:37.034177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 520/1048] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:44:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:44:37.038294 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e985e2-ffcb-4b27-b4c2-84d6a579ff29 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:37.070172 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7185ee0b-a086-4e10-9a3a-73afa94425d2 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:37.104240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a2f246-443b-4e74-913d-11f146876243 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:37.104863 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a2f246-443b-4e74-913d-11f146876243 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:37.105224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a2f246-443b-4e74-913d-11f146876243 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:37.105562 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a2f246-443b-4e74-913d-11f146876243 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:37.141717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e985e2-ffcb-4b27-b4c2-84d6a579ff29 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:37.175083 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a2f246-443b-4e74-913d-11f146876243 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:37.175381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9b7349-db21-4366-88c9-00d6fef9eec4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:37.175979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9b7349-db21-4366-88c9-00d6fef9eec4 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:37.176252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9b7349-db21-4366-88c9-00d6fef9eec4 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:37.176517 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9b7349-db21-4366-88c9-00d6fef9eec4 None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:37.176613 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a2f246-443b-4e74-913d-11f146876243 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:37.177215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9b7349-db21-4366-88c9-00d6fef9eec4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:37.177492 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9b7349-db21-4366-88c9-00d6fef9eec4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:37.178922 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9b7349-db21-4366-88c9-00d6fef9eec4 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:37.190712 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 521/1049] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:44:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:37.196832 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa172b6-4e9a-46de-9137-08ddac15f3a1 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:37.197114 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa172b6-4e9a-46de-9137-08ddac15f3a1 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:37.197397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa172b6-4e9a-46de-9137-08ddac15f3a1 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:37.198637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 522/1050] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:37.199320 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa172b6-4e9a-46de-9137-08ddac15f3a1 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:37.213297 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:37.220036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:37.288730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 529/1051] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:44:36 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 23 13:44:37.288730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d275649e-a539-4ed6-8adf-9f065253c0fa None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:37.288730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d275649e-a539-4ed6-8adf-9f065253c0fa None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:37.288730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d275649e-a539-4ed6-8adf-9f065253c0fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:37.301292 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa172b6-4e9a-46de-9137-08ddac15f3a1 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:37.305018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-d275649e-a539-4ed6-8adf-9f065253c0fa None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:37.306190 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d275649e-a539-4ed6-8adf-9f065253c0fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:37.334789 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840c21c7-ef7d-4ff9-8e48-3d69b8055dc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:37.335385 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840c21c7-ef7d-4ff9-8e48-3d69b8055dc7 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:37.335657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840c21c7-ef7d-4ff9-8e48-3d69b8055dc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:37.335920 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840c21c7-ef7d-4ff9-8e48-3d69b8055dc7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:37.336635 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840c21c7-ef7d-4ff9-8e48-3d69b8055dc7 None admin] RBAC: Adding query filter params (role.id=b2084c06d5a44297a9a7795d559fae94, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=f01ed94dbb384a178ad0a45a6f3d8c17) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:37.336936 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840c21c7-ef7d-4ff9-8e48-3d69b8055dc7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:37.338346 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840c21c7-ef7d-4ff9-8e48-3d69b8055dc7 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:37.347214 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 523/1052] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:44:37 2022] GET /identity/v3/role_assignments?user.id=f01ed94dbb384a178ad0a45a6f3d8c17&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=b2084c06d5a44297a9a7795d559fae94 => generated 259 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:37.348000 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840c21c7-ef7d-4ff9-8e48-3d69b8055dc7 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:37.384390 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 530/1053] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:37 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:37.432631 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:37.458360 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840c21c7-ef7d-4ff9-8e48-3d69b8055dc7 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:37.496725 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8864e85e-2f91-4dbe-a5bd-f5e84db7315d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:37.497731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8864e85e-2f91-4dbe-a5bd-f5e84db7315d None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:37.498238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8864e85e-2f91-4dbe-a5bd-f5e84db7315d None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:37.498777 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8864e85e-2f91-4dbe-a5bd-f5e84db7315d None admin] PATH_INFO: `/v3/projects/65ee0f795c8b406ea7bc6f5ad7f3e336` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:37.500177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8864e85e-2f91-4dbe-a5bd-f5e84db7315d None admin] RBAC: Authorizing `identity:get_project(project_id=65ee0f795c8b406ea7bc6f5ad7f3e336)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:37.501848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8864e85e-2f91-4dbe-a5bd-f5e84db7315d None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:37.504101 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 524/1054] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:44:37 2022] GET /identity/v3/projects/65ee0f795c8b406ea7bc6f5ad7f3e336 => generated 311 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:37.509960 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d275649e-a539-4ed6-8adf-9f065253c0fa None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:37.577611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:37.612191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d275649e-a539-4ed6-8adf-9f065253c0fa None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:37.648353 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88eb8a03-7b67-48e5-acbb-fdde821979c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:37.648880 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eb8a03-7b67-48e5-acbb-fdde821979c8 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:37.649162 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eb8a03-7b67-48e5-acbb-fdde821979c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:37.649433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eb8a03-7b67-48e5-acbb-fdde821979c8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:37.650157 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88eb8a03-7b67-48e5-acbb-fdde821979c8 None admin] RBAC: Adding query filter params (role.id=727f0fa598e54b9a929d2cf6277c875e, scope.project.id=65ee0f795c8b406ea7bc6f5ad7f3e336, user.id=68fae1e50b4347b692dc718993d6fe73) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:37.650836 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88eb8a03-7b67-48e5-acbb-fdde821979c8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:37.651588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8864e85e-2f91-4dbe-a5bd-f5e84db7315d None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:37.652267 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88eb8a03-7b67-48e5-acbb-fdde821979c8 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:37.666825 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 531/1055] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:44:37 2022] GET /identity/v3/role_assignments?user.id=68fae1e50b4347b692dc718993d6fe73&scope.project.id=65ee0f795c8b406ea7bc6f5ad7f3e336&role.id=727f0fa598e54b9a929d2cf6277c875e => generated 259 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:37.766415 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8864e85e-2f91-4dbe-a5bd-f5e84db7315d None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:37.803536 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54316968-aef9-4c0a-a1a3-4a4a023dd701 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:37.804282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54316968-aef9-4c0a-a1a3-4a4a023dd701 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:37.804639 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54316968-aef9-4c0a-a1a3-4a4a023dd701 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:37.804981 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54316968-aef9-4c0a-a1a3-4a4a023dd701 None admin] PATH_INFO: `/v3/services/image` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:37.805649 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54316968-aef9-4c0a-a1a3-4a4a023dd701 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:37.807145 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54316968-aef9-4c0a-a1a3-4a4a023dd701 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:37.817152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-54316968-aef9-4c0a-a1a3-4a4a023dd701 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 23 13:44:37.817152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:37.817152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:44:37.817152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:44:37.817152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:44:37.817152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:44:37.817152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:44:37.817152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:37.817152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 23 13:44:37.817152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:37.817152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:44:37.817152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:37.817152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:37.817152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:37.817152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:37.817152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:37.817152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:37.818439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:37.818439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:37.818439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:37.818439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:37.818439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:37.818439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:37.818439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:44:37.818439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:44:37.818439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:44:37.818439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:44:37.818439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:37.818439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:37.818439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:37.818439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:37.818439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:37.818439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:37.819471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:37.819471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:37.819471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:37.819471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:37.819471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:37.819471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:37.819471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:37.819471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:37.819471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:37.819471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:37.819471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:44:37.819471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:44:37.819471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 23 13:44:37.819471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:37.825327 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 525/1056] 10.222.0.62 () {56 vars in 1105 bytes} [Fri Sep 23 13:44:37 2022] GET /identity/v3/services/image => generated 86 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:44:37.826844 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88eb8a03-7b67-48e5-acbb-fdde821979c8 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:37.933601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88eb8a03-7b67-48e5-acbb-fdde821979c8 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:37.962043 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:37.968105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c2bab1-4f80-4df8-a33a-483123992318 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:37.968739 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c2bab1-4f80-4df8-a33a-483123992318 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:37.969067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c2bab1-4f80-4df8-a33a-483123992318 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:37.969422 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c2bab1-4f80-4df8-a33a-483123992318 None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:37.970059 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c2bab1-4f80-4df8-a33a-483123992318 None admin] RBAC: Adding query filter params (name=image) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:37.981758 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c2bab1-4f80-4df8-a33a-483123992318 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:37.981758 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c2bab1-4f80-4df8-a33a-483123992318 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:37.981758 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 532/1057] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:37 2022] GET /identity/v3/services?name=image => generated 122 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:37.982414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54316968-aef9-4c0a-a1a3-4a4a023dd701 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:37.992972 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:38.087909 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54316968-aef9-4c0a-a1a3-4a4a023dd701 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:38.123194 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bce51f6-e25c-4968-924b-3f25aae6384a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:38.123803 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bce51f6-e25c-4968-924b-3f25aae6384a None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:38.124082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bce51f6-e25c-4968-924b-3f25aae6384a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:38.124353 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bce51f6-e25c-4968-924b-3f25aae6384a None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:38.125005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bce51f6-e25c-4968-924b-3f25aae6384a None admin] RBAC: Adding query filter params (type=image) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:38.125366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bce51f6-e25c-4968-924b-3f25aae6384a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:38.126765 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bce51f6-e25c-4968-924b-3f25aae6384a None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:38.132678 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 526/1058] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:37 2022] GET /identity/v3/services?type=image => generated 350 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:38.137560 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c2bab1-4f80-4df8-a33a-483123992318 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:38.252358 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c2bab1-4f80-4df8-a33a-483123992318 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:38.287544 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ecbcee-17c7-44ac-9965-1dab626a408f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:38.288152 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ecbcee-17c7-44ac-9965-1dab626a408f None admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:38.288429 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ecbcee-17c7-44ac-9965-1dab626a408f None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:38.288697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ecbcee-17c7-44ac-9965-1dab626a408f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:38.289352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ecbcee-17c7-44ac-9965-1dab626a408f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:38.290734 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ecbcee-17c7-44ac-9965-1dab626a408f None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:38.398903 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 533/1059] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 13:44:38 2022] POST /identity/v3/endpoints => generated 327 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 13:44:39.540472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bea87e-0cc4-46db-9d28-2501e4ecd0a0 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:39.540979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bea87e-0cc4-46db-9d28-2501e4ecd0a0 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:39.540979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bea87e-0cc4-46db-9d28-2501e4ecd0a0 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:39.542449 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 527/1060] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:39.547447 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4017d2-0f44-4621-b904-e636133901be None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:39.547839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4017d2-0f44-4621-b904-e636133901be None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:39.548031 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4017d2-0f44-4621-b904-e636133901be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:39.566468 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-da4017d2-0f44-4621-b904-e636133901be None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:39.567644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-da4017d2-0f44-4621-b904-e636133901be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:39.614287 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:39.641484 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 534/1061] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:39.847240 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c867c5bf-d02a-45d5-a3fc-6028b135a15d None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:39.847435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c867c5bf-d02a-45d5-a3fc-6028b135a15d None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:39.847689 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c867c5bf-d02a-45d5-a3fc-6028b135a15d None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:39.849173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 528/1062] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:39.854218 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a709cca4-fc23-4288-b59a-91a2bbc362b2 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:39.854547 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a709cca4-fc23-4288-b59a-91a2bbc362b2 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:39.854817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a709cca4-fc23-4288-b59a-91a2bbc362b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:39.873428 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-a709cca4-fc23-4288-b59a-91a2bbc362b2 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:39.874635 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a709cca4-fc23-4288-b59a-91a2bbc362b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:39.911032 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c867c5bf-d02a-45d5-a3fc-6028b135a15d None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:39.943544 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 535/1063] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:39.988689 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:40.020732 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c867c5bf-d02a-45d5-a3fc-6028b135a15d None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:40.055190 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cf5fd6-8b4b-4d1b-8752-8cf71ee92342 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:40.055808 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cf5fd6-8b4b-4d1b-8752-8cf71ee92342 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:40.056081 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cf5fd6-8b4b-4d1b-8752-8cf71ee92342 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:40.056345 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cf5fd6-8b4b-4d1b-8752-8cf71ee92342 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:40.069641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6cf5fd6-8b4b-4d1b-8752-8cf71ee92342 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:44:40.074397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-b6cf5fd6-8b4b-4d1b-8752-8cf71ee92342 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 23 13:44:40.074397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:40.074397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:40.074397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:40.074397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:40.074397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:40.074397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:40.074397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:40.074397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:40.074397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:40.074397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:40.074397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:40.074397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:44:40.074397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:44:40.074397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:44:40.074397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:44:40.075308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:44:40.075308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:44:40.075308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:44:40.075308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:44:40.075308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:40.075308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:40.075308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:40.075308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:40.075308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:40.075308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:40.075308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:40.075308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:40.075308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:40.075308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:40.075308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:40.075308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:40.076153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:40.076153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:40.076153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:40.076153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:40.076153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:44:40.076153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:44:40.076153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:44:40.076153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:44:40.076153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:44:40.076153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:44:40.076153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 23 13:44:40.076153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:40.077420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 529/1064] 10.222.0.62 () {56 vars in 1120 bytes} [Fri Sep 23 13:44:39 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:40.081412 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a709cca4-fc23-4288-b59a-91a2bbc362b2 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:40.182657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a709cca4-fc23-4288-b59a-91a2bbc362b2 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:40.190472 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:40.225254 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e9ca68-e5ca-4ddd-bb32-f012aee8a3ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:40.225905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e9ca68-e5ca-4ddd-bb32-f012aee8a3ae None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:40.226617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e9ca68-e5ca-4ddd-bb32-f012aee8a3ae None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:40.227253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e9ca68-e5ca-4ddd-bb32-f012aee8a3ae None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:40.228408 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e9ca68-e5ca-4ddd-bb32-f012aee8a3ae None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:40.228706 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e9ca68-e5ca-4ddd-bb32-f012aee8a3ae None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:40.230063 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e9ca68-e5ca-4ddd-bb32-f012aee8a3ae None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:40.231644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cf5fd6-8b4b-4d1b-8752-8cf71ee92342 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:40.244863 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 536/1065] 10.222.0.62 () {56 vars in 1115 bytes} [Fri Sep 23 13:44:40 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:40.246507 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e9ca68-e5ca-4ddd-bb32-f012aee8a3ae None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:40.339225 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cf5fd6-8b4b-4d1b-8752-8cf71ee92342 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:40.356152 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e9ca68-e5ca-4ddd-bb32-f012aee8a3ae None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:40.372280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:40.383043 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2a207a-0d10-44fa-a646-ba5022b53d5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:40.390825 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2a207a-0d10-44fa-a646-ba5022b53d5b None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:40.390825 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2a207a-0d10-44fa-a646-ba5022b53d5b None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:40.390825 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2a207a-0d10-44fa-a646-ba5022b53d5b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:40.391260 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0f4184-f49d-4f95-892d-4cb4aa6622de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:40.391897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0f4184-f49d-4f95-892d-4cb4aa6622de None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:40.392173 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0f4184-f49d-4f95-892d-4cb4aa6622de None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:40.392434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0f4184-f49d-4f95-892d-4cb4aa6622de None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:40.396494 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2a207a-0d10-44fa-a646-ba5022b53d5b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:44:40.401747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-9e2a207a-0d10-44fa-a646-ba5022b53d5b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:44:40.401747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:40.401747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:40.401747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:40.401747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:40.401747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:40.401747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:40.401747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:40.401747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:40.401747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:40.401747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:40.401747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:40.401747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:44:40.401747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:44:40.401747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:44:40.401747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:44:40.402701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:44:40.402701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:44:40.402701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:44:40.402701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:44:40.402701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:40.402701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:40.402701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:40.402701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:40.402701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:40.402701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:40.402701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:40.402701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:40.402701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:40.402701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:40.402701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:40.402701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:40.403598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:40.403598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:40.403598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:40.403598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:40.403598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:44:40.403598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:44:40.403598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:44:40.403598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:44:40.403598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:44:40.403598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:44:40.403598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:44:40.403598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:40.403598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 530/1066] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:44:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:44:40.405834 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0f4184-f49d-4f95-892d-4cb4aa6622de None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:40.407909 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0f4184-f49d-4f95-892d-4cb4aa6622de None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:40.411015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2a207a-0d10-44fa-a646-ba5022b53d5b None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:40.425650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-0c0f4184-f49d-4f95-892d-4cb4aa6622de None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:40.425650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:40.425650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:40.425650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:40.425650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:40.425650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:40.425650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:40.425650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:40.425650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:40.425650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:40.425650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:40.425650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:40.425650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:40.425650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:40.425650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:40.425650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:40.426817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:40.426817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:40.426817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:40.426817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:40.426817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:40.426817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:40.426817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:40.426817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:40.426817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:40.426817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:40.426817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:40.426817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:40.426817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:40.426817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:40.426817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:40.426817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:40.426817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:40.427885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:40.427885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:40.427885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:40.429273 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 537/1067] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:44:40.434914 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0f4184-f49d-4f95-892d-4cb4aa6622de None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:40.458118 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:40.524306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2a207a-0d10-44fa-a646-ba5022b53d5b None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:40.542993 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0f4184-f49d-4f95-892d-4cb4aa6622de None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:40.560650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b86982-cd23-4ab0-842f-85e112598fce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:40.561256 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b86982-cd23-4ab0-842f-85e112598fce None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:40.561531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b86982-cd23-4ab0-842f-85e112598fce None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:40.561796 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b86982-cd23-4ab0-842f-85e112598fce None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:40.562669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b86982-cd23-4ab0-842f-85e112598fce None admin] RBAC: Adding query filter params (name=admin) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:40.562970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b86982-cd23-4ab0-842f-85e112598fce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:40.564315 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b86982-cd23-4ab0-842f-85e112598fce None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:40.577229 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 531/1068] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:44:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:40.578272 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b86982-cd23-4ab0-842f-85e112598fce None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:40.586121 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd82f5dd-3e1c-456d-99da-f155bc6b0afc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:40.586781 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd82f5dd-3e1c-456d-99da-f155bc6b0afc None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:40.587156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd82f5dd-3e1c-456d-99da-f155bc6b0afc None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:40.587466 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd82f5dd-3e1c-456d-99da-f155bc6b0afc None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:40.588085 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd82f5dd-3e1c-456d-99da-f155bc6b0afc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:40.588385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd82f5dd-3e1c-456d-99da-f155bc6b0afc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:40.589739 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd82f5dd-3e1c-456d-99da-f155bc6b0afc None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:40.606065 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 538/1069] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:40.607221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd82f5dd-3e1c-456d-99da-f155bc6b0afc None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:40.637343 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:40.684283 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b86982-cd23-4ab0-842f-85e112598fce None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:40.690404 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:40.708916 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd82f5dd-3e1c-456d-99da-f155bc6b0afc None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:40.718080 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d49d2f3-e69e-424c-a754-a7fab89140a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:40.719383 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d49d2f3-e69e-424c-a754-a7fab89140a0 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:40.719888 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d49d2f3-e69e-424c-a754-a7fab89140a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:40.720173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d49d2f3-e69e-424c-a754-a7fab89140a0 None admin] PATH_INFO: `/v3/users/68fae1e50b4347b692dc718993d6fe73` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:40.721318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d49d2f3-e69e-424c-a754-a7fab89140a0 None admin] RBAC: Authorizing `identity:get_user(user_id=68fae1e50b4347b692dc718993d6fe73)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:40.722782 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d49d2f3-e69e-424c-a754-a7fab89140a0 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:40.724369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 532/1070] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:44:40 2022] GET /identity/v3/users/68fae1e50b4347b692dc718993d6fe73 => generated 285 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:40.725492 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a255f9e-4666-4425-8604-8882a07f5acb None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:40.725771 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a255f9e-4666-4425-8604-8882a07f5acb None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:40.726044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a255f9e-4666-4425-8604-8882a07f5acb None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:40.730698 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 533/1071] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:40 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:40.730698 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a255f9e-4666-4425-8604-8882a07f5acb None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:40.745855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719d5e3c-8d5b-4069-b6dd-a6c834254f2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:40.746518 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719d5e3c-8d5b-4069-b6dd-a6c834254f2a None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:40.746797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719d5e3c-8d5b-4069-b6dd-a6c834254f2a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:40.747067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719d5e3c-8d5b-4069-b6dd-a6c834254f2a None admin] PATH_INFO: `/v3/users/nova` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:40.754757 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719d5e3c-8d5b-4069-b6dd-a6c834254f2a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:40.756160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-719d5e3c-8d5b-4069-b6dd-a6c834254f2a None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:40.767657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-719d5e3c-8d5b-4069-b6dd-a6c834254f2a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 23 13:44:40.767657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:40.767657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:40.767657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:40.767657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:40.767657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:40.767657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:40.767657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:40.767657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:40.767657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:40.767657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:40.767657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:40.767657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:44:40.767657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:44:40.767657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:44:40.767657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:44:40.768624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:40.768624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:40.768624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:44:40.768624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:40.768624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:44:40.768624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:40.768624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:40.768624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:40.768624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:40.768624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:40.768624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:40.768624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:40.768624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:40.768624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:40.768624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:40.768624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:40.768624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:40.768624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:40.770023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:40.770023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:40.770023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:44:40.770023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:44:40.770023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:44:40.770023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:44:40.770023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:44:40.770023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:44:40.770023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 23 13:44:40.770023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:40.770023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 539/1072] 10.222.0.62 () {56 vars in 1093 bytes} [Fri Sep 23 13:44:40 2022] GET /identity/v3/users/nova => generated 82 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:40.770762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c648249a-91b4-4a1c-9311-d48c8c1dcbc0 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:40.770762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c648249a-91b4-4a1c-9311-d48c8c1dcbc0 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:40.770762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c648249a-91b4-4a1c-9311-d48c8c1dcbc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:40.787912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c648249a-91b4-4a1c-9311-d48c8c1dcbc0 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:40.789091 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c648249a-91b4-4a1c-9311-d48c8c1dcbc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:40.834811 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a255f9e-4666-4425-8604-8882a07f5acb None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:40.854092 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:40.855516 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 540/1073] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:40.856491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c648249a-91b4-4a1c-9311-d48c8c1dcbc0 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:40.885009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf572152-ec06-42a8-b5d5-473b1ce09b02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:40.885894 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf572152-ec06-42a8-b5d5-473b1ce09b02 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:40.886176 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf572152-ec06-42a8-b5d5-473b1ce09b02 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:40.886487 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf572152-ec06-42a8-b5d5-473b1ce09b02 None admin] PATH_INFO: `/v3/projects/65ee0f795c8b406ea7bc6f5ad7f3e336` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:40.887677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf572152-ec06-42a8-b5d5-473b1ce09b02 None admin] RBAC: Authorizing `identity:get_project(project_id=65ee0f795c8b406ea7bc6f5ad7f3e336)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:40.889575 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf572152-ec06-42a8-b5d5-473b1ce09b02 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:40.897371 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 534/1074] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:44:40 2022] GET /identity/v3/projects/65ee0f795c8b406ea7bc6f5ad7f3e336 => generated 311 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:40.898111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf572152-ec06-42a8-b5d5-473b1ce09b02 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:40.926979 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:40.957845 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c648249a-91b4-4a1c-9311-d48c8c1dcbc0 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:40.993796 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c33d14-5a87-4bda-95e6-9e778e8c3218 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:40.994434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c33d14-5a87-4bda-95e6-9e778e8c3218 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:40.994713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c33d14-5a87-4bda-95e6-9e778e8c3218 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:40.994988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c33d14-5a87-4bda-95e6-9e778e8c3218 None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:40.995709 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c33d14-5a87-4bda-95e6-9e778e8c3218 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:40.996015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c33d14-5a87-4bda-95e6-9e778e8c3218 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:40.997432 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c33d14-5a87-4bda-95e6-9e778e8c3218 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:41.003890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf572152-ec06-42a8-b5d5-473b1ce09b02 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:41.020116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 541/1075] 10.222.0.62 () {56 vars in 1133 bytes} [Fri Sep 23 13:44:40 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:41.021155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c33d14-5a87-4bda-95e6-9e778e8c3218 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:41.046255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfea57d-1faf-4bfd-ba56-c7249764dc88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:41.047010 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfea57d-1faf-4bfd-ba56-c7249764dc88 None admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:41.047317 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfea57d-1faf-4bfd-ba56-c7249764dc88 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:41.047591 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfea57d-1faf-4bfd-ba56-c7249764dc88 None admin] PATH_INFO: `/v3/projects/65ee0f795c8b406ea7bc6f5ad7f3e336/users/68fae1e50b4347b692dc718993d6fe73/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:41.049612 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfea57d-1faf-4bfd-ba56-c7249764dc88 None admin] RBAC: Authorizing `identity:create_grant(project_id=65ee0f795c8b406ea7bc6f5ad7f3e336, user_id=68fae1e50b4347b692dc718993d6fe73, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:41.051154 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfea57d-1faf-4bfd-ba56-c7249764dc88 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:41.122431 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c33d14-5a87-4bda-95e6-9e778e8c3218 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:41.131641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 535/1076] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:44:40 2022] PUT /identity/v3/projects/65ee0f795c8b406ea7bc6f5ad7f3e336/users/68fae1e50b4347b692dc718993d6fe73/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 13:44:41.132314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfea57d-1faf-4bfd-ba56-c7249764dc88 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:41.156148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:41.165039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ef9292-52c9-4672-9aed-ab7e98edb5d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:41.166058 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ef9292-52c9-4672-9aed-ab7e98edb5d8 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:41.166370 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ef9292-52c9-4672-9aed-ab7e98edb5d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:41.166646 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ef9292-52c9-4672-9aed-ab7e98edb5d8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:41.178630 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ef9292-52c9-4672-9aed-ab7e98edb5d8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:41.180092 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ef9292-52c9-4672-9aed-ab7e98edb5d8 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:41.198960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:41.201776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-48ef9292-52c9-4672-9aed-ab7e98edb5d8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:41.201776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:41.201776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:41.201776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:41.201776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:41.201776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:41.201776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:41.201776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:41.201776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:41.201776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:41.201776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:41.201776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:41.201776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:41.201776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:41.201776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:41.201776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:41.202870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:41.202870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:41.202870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:41.202870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:41.202870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:41.202870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:41.202870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:41.202870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:41.202870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:41.202870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:41.202870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:41.202870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:41.202870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:41.202870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:41.202870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:41.202870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:41.202870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:41.204473 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:41.204473 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:41.204473 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:41.212098 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 542/1077] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:41.215550 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ef9292-52c9-4672-9aed-ab7e98edb5d8 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:41.266499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfea57d-1faf-4bfd-ba56-c7249764dc88 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:41.306123 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8423225c-fcef-4f8a-9735-4930bd2a1208 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:41.306769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8423225c-fcef-4f8a-9735-4930bd2a1208 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:41.307049 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8423225c-fcef-4f8a-9735-4930bd2a1208 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:41.307335 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8423225c-fcef-4f8a-9735-4930bd2a1208 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:41.337767 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ef9292-52c9-4672-9aed-ab7e98edb5d8 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:41.376658 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1665e205-39e0-4885-b4c0-f43e3d35beee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:41.377264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665e205-39e0-4885-b4c0-f43e3d35beee None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:41.377539 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665e205-39e0-4885-b4c0-f43e3d35beee None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:41.377805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1665e205-39e0-4885-b4c0-f43e3d35beee None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:41.378478 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1665e205-39e0-4885-b4c0-f43e3d35beee None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:41.378783 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1665e205-39e0-4885-b4c0-f43e3d35beee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:41.380128 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1665e205-39e0-4885-b4c0-f43e3d35beee None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:41.382253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8423225c-fcef-4f8a-9735-4930bd2a1208 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:41.383707 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8423225c-fcef-4f8a-9735-4930bd2a1208 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:41.393296 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 543/1078] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:41.398816 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1665e205-39e0-4885-b4c0-f43e3d35beee None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:41.434783 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:41.443319 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:41.489373 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 536/1079] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:44:41 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 23 13:44:41.495085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8423225c-fcef-4f8a-9735-4930bd2a1208 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:41.503286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1665e205-39e0-4885-b4c0-f43e3d35beee None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:41.539661 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6edc325-0fc8-4740-8e03-a76c1e568da5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:41.540284 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6edc325-0fc8-4740-8e03-a76c1e568da5 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:41.540559 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6edc325-0fc8-4740-8e03-a76c1e568da5 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:41.540822 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6edc325-0fc8-4740-8e03-a76c1e568da5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:41.548069 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6edc325-0fc8-4740-8e03-a76c1e568da5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:41.549525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6edc325-0fc8-4740-8e03-a76c1e568da5 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:41.560713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-a6edc325-0fc8-4740-8e03-a76c1e568da5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:41.560713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:41.560713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:41.560713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:41.560713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:41.560713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:41.560713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:41.560713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:41.560713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:41.560713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:41.560713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:41.560713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:41.560713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:44:41.560713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:44:41.560713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:44:41.561849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:44:41.561849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:41.561849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:41.561849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:41.561849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:41.561849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:41.561849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:41.561849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:41.561849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:41.561849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:41.561849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:41.561849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:41.561849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:41.561849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:41.561849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:41.561849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:41.561849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:41.563708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:44:41.563708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:44:41.563708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:44:41.563708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:44:41.563708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:44:41.563708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:44:41.563708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:41.563708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:41.567343 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 544/1080] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:44:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:44:41.568280 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6edc325-0fc8-4740-8e03-a76c1e568da5 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:41.616582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8423225c-fcef-4f8a-9735-4930bd2a1208 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:41.651590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5946629-2ae2-4956-81ab-40633bd664c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:41.652252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5946629-2ae2-4956-81ab-40633bd664c4 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:41.652487 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5946629-2ae2-4956-81ab-40633bd664c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:41.652751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5946629-2ae2-4956-81ab-40633bd664c4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:41.663485 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5946629-2ae2-4956-81ab-40633bd664c4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:41.664910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5946629-2ae2-4956-81ab-40633bd664c4 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:41.669347 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:41.672810 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6edc325-0fc8-4740-8e03-a76c1e568da5 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:41.674729 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:41.681724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-e5946629-2ae2-4956-81ab-40633bd664c4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:41.681724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:41.681724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:41.681724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:41.681724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:41.681724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:41.681724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:41.681724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:41.681724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:41.681724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:41.681724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:41.681724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:41.681724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:41.681724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:41.681724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:41.681724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:41.687062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:41.687062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:41.687062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:41.687062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:41.687062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:41.687062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:41.687062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:41.687062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:41.687062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:41.687062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:41.687062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:41.687062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:41.687062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:41.687062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:41.687062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:41.687062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:41.687062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:41.688529 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:41.688529 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:41.688529 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:41.688529 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 537/1081] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:44:41.690475 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5946629-2ae2-4956-81ab-40633bd664c4 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:41.709921 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca0af79-9970-45c4-8fd8-06aadeb266b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:41.710557 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca0af79-9970-45c4-8fd8-06aadeb266b7 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:41.710832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca0af79-9970-45c4-8fd8-06aadeb266b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:41.711095 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca0af79-9970-45c4-8fd8-06aadeb266b7 None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:41.711775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca0af79-9970-45c4-8fd8-06aadeb266b7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:41.712076 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca0af79-9970-45c4-8fd8-06aadeb266b7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:41.713481 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca0af79-9970-45c4-8fd8-06aadeb266b7 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:41.730676 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 545/1082] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:44:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:41.732422 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca0af79-9970-45c4-8fd8-06aadeb266b7 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:41.804927 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5946629-2ae2-4956-81ab-40633bd664c4 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:41.840923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca0af79-9970-45c4-8fd8-06aadeb266b7 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:41.841437 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e70812a-a59f-4b80-9a9f-a36bc9906de5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:41.842072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e70812a-a59f-4b80-9a9f-a36bc9906de5 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:41.842383 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e70812a-a59f-4b80-9a9f-a36bc9906de5 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:41.842643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e70812a-a59f-4b80-9a9f-a36bc9906de5 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:41.844442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e70812a-a59f-4b80-9a9f-a36bc9906de5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:41.844763 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e70812a-a59f-4b80-9a9f-a36bc9906de5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:41.846137 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e70812a-a59f-4b80-9a9f-a36bc9906de5 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:41.866430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 538/1083] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:41.878256 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d3830c9-0b49-4d3c-84a2-93aa9586395d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:41.879056 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3830c9-0b49-4d3c-84a2-93aa9586395d None admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:41.879399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3830c9-0b49-4d3c-84a2-93aa9586395d None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:41.879732 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d3830c9-0b49-4d3c-84a2-93aa9586395d None admin] PATH_INFO: `/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/f01ed94dbb384a178ad0a45a6f3d8c17/roles/b2084c06d5a44297a9a7795d559fae94` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:41.882103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3830c9-0b49-4d3c-84a2-93aa9586395d None admin] RBAC: Authorizing `identity:create_grant(project_id=2a19fd4315ad4a6eb349dfe8cfa66c23, user_id=f01ed94dbb384a178ad0a45a6f3d8c17, role_id=b2084c06d5a44297a9a7795d559fae94)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:41.883672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d3830c9-0b49-4d3c-84a2-93aa9586395d None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:41.976788 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 546/1084] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:44:41 2022] PUT /identity/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/f01ed94dbb384a178ad0a45a6f3d8c17/roles/b2084c06d5a44297a9a7795d559fae94 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 13:44:43.323185 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f854184-4d93-420d-bbb5-743483a4ffae None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:43.323719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f854184-4d93-420d-bbb5-743483a4ffae None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:43.323719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f854184-4d93-420d-bbb5-743483a4ffae None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:43.325058 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 539/1085] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:43.330516 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b512ec-87fd-4498-8cdd-70a7cb6f3238 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:43.330922 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b512ec-87fd-4498-8cdd-70a7cb6f3238 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:43.331184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b512ec-87fd-4498-8cdd-70a7cb6f3238 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:43.350068 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-48b512ec-87fd-4498-8cdd-70a7cb6f3238 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:43.351284 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-48b512ec-87fd-4498-8cdd-70a7cb6f3238 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:43.366313 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:43.416450 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 547/1086] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:43.683840 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f854184-4d93-420d-bbb5-743483a4ffae None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:43.742727 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:43.783518 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f854184-4d93-420d-bbb5-743483a4ffae None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:43.816483 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06131777-51b0-4f28-845c-22964110e3c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:43.817106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06131777-51b0-4f28-845c-22964110e3c1 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:43.817381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06131777-51b0-4f28-845c-22964110e3c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:43.817648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06131777-51b0-4f28-845c-22964110e3c1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:43.827414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06131777-51b0-4f28-845c-22964110e3c1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:44:43.832640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-06131777-51b0-4f28-845c-22964110e3c1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:44:43.832640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:43.832640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:43.832640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:43.832640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:43.832640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:43.832640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:43.832640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:43.832640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:43.832640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:43.832640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:43.832640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:43.832640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:44:43.832640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:44:43.832640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:44:43.832640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:44:43.833596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:44:43.833596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:44:43.833596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:44:43.833596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:44:43.833596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:43.833596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:43.833596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:43.833596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:43.833596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:43.833596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:43.833596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:43.833596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:43.833596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:43.833596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:43.833596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:43.833596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:43.834466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:43.834466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:43.834466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:43.834466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:43.834466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:44:43.834466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:44:43.834466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:44:43.834466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:44:43.834466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:44:43.834466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:44:43.834466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:44:43.834466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:43.834466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 540/1087] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:44:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:43.839690 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b512ec-87fd-4498-8cdd-70a7cb6f3238 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:43.940002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b512ec-87fd-4498-8cdd-70a7cb6f3238 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:43.975546 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988694d6-69ec-405e-8d53-1e7646f68243 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:43.976145 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988694d6-69ec-405e-8d53-1e7646f68243 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:43.976425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988694d6-69ec-405e-8d53-1e7646f68243 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:43.976694 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988694d6-69ec-405e-8d53-1e7646f68243 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:43.977350 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988694d6-69ec-405e-8d53-1e7646f68243 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:43.977649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988694d6-69ec-405e-8d53-1e7646f68243 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:43.979033 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988694d6-69ec-405e-8d53-1e7646f68243 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:43.987255 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 548/1088] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:44:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:43.993666 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06131777-51b0-4f28-845c-22964110e3c1 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:44.099069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06131777-51b0-4f28-845c-22964110e3c1 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:44.119633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:44.122657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ae3691-6892-40b4-8be6-bead9bf6dc33 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:44.128513 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ae3691-6892-40b4-8be6-bead9bf6dc33 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:44.128513 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ae3691-6892-40b4-8be6-bead9bf6dc33 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:44.128513 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 549/1089] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:44.130243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec16a93-20d4-4858-b982-c424d9667e42 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:44.130553 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec16a93-20d4-4858-b982-c424d9667e42 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:44.130827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec16a93-20d4-4858-b982-c424d9667e42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:44.134041 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c86a07-2ea8-4a42-8e65-ad15e112f286 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:44.134688 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c86a07-2ea8-4a42-8e65-ad15e112f286 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:44.134962 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c86a07-2ea8-4a42-8e65-ad15e112f286 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:44.135228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c86a07-2ea8-4a42-8e65-ad15e112f286 None admin] PATH_INFO: `/v3/users/68fae1e50b4347b692dc718993d6fe73` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:44.136326 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c86a07-2ea8-4a42-8e65-ad15e112f286 None admin] RBAC: Authorizing `identity:get_user(user_id=68fae1e50b4347b692dc718993d6fe73)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:44.137732 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c86a07-2ea8-4a42-8e65-ad15e112f286 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:44.139389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 541/1090] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:44:43 2022] GET /identity/v3/users/68fae1e50b4347b692dc718993d6fe73 => generated 285 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:44.146168 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c86a07-2ea8-4a42-8e65-ad15e112f286 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:44.149192 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-9ec16a93-20d4-4858-b982-c424d9667e42 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:44.150496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-9ec16a93-20d4-4858-b982-c424d9667e42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:44.157883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:44.221273 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 550/1091] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 23 13:44:44.248297 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c86a07-2ea8-4a42-8e65-ad15e112f286 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:44.284076 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f4c25e-9370-4fbe-be47-7c70fbbe5191 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:44.284679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f4c25e-9370-4fbe-be47-7c70fbbe5191 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:44.284957 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f4c25e-9370-4fbe-be47-7c70fbbe5191 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:44.285224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f4c25e-9370-4fbe-be47-7c70fbbe5191 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:44.339169 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:44.350999 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f4c25e-9370-4fbe-be47-7c70fbbe5191 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:44.352462 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f4c25e-9370-4fbe-be47-7c70fbbe5191 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:44.506484 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 542/1092] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:44:44 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 23 13:44:44.512250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f4c25e-9370-4fbe-be47-7c70fbbe5191 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:44.512430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec16a93-20d4-4858-b982-c424d9667e42 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:44.599266 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:44.611553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f4c25e-9370-4fbe-be47-7c70fbbe5191 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:44.615755 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec16a93-20d4-4858-b982-c424d9667e42 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:44.646964 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f807d4-e155-4a60-8013-52811488053f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:44.647576 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f807d4-e155-4a60-8013-52811488053f None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:44.647851 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f807d4-e155-4a60-8013-52811488053f None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:44.648116 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f807d4-e155-4a60-8013-52811488053f None admin] PATH_INFO: `/v3/projects/65ee0f795c8b406ea7bc6f5ad7f3e336` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:44.649196 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f807d4-e155-4a60-8013-52811488053f None admin] RBAC: Authorizing `identity:get_project(project_id=65ee0f795c8b406ea7bc6f5ad7f3e336)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:44.650632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f807d4-e155-4a60-8013-52811488053f None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:44.651388 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7232c6ed-41de-46cd-b80e-fec2df4613de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:44.652015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7232c6ed-41de-46cd-b80e-fec2df4613de None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:44.652256 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 543/1093] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:44:44 2022] GET /identity/v3/projects/65ee0f795c8b406ea7bc6f5ad7f3e336 => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:44.652393 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7232c6ed-41de-46cd-b80e-fec2df4613de None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:44.652554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7232c6ed-41de-46cd-b80e-fec2df4613de None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:44.658907 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f807d4-e155-4a60-8013-52811488053f None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:44.663417 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7232c6ed-41de-46cd-b80e-fec2df4613de None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:44:44.668158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-7232c6ed-41de-46cd-b80e-fec2df4613de None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 23 13:44:44.668158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:44.668158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:44.668158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:44.668158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:44.668158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:44.668158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:44.668158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:44.668158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:44.668158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:44.668158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:44.668158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:44.668158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:44:44.668158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:44:44.668158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:44:44.668158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:44:44.669198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:44:44.669198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:44:44.669198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:44:44.669198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:44:44.669198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:44.669198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:44.669198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:44.669198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:44.669198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:44.669198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:44.669198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:44.669198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:44.669198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:44.669198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:44.669198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:44.669198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:44.670155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:44.670155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:44.670155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:44.670155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:44.670155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:44:44.670155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:44:44.670155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:44:44.670155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:44:44.670155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:44:44.670155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:44:44.670155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 23 13:44:44.670155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:44.670155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 551/1094] 10.222.0.62 () {56 vars in 1120 bytes} [Fri Sep 23 13:44:44 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:44:44.675887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7232c6ed-41de-46cd-b80e-fec2df4613de None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:44.679173 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:44.794689 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f807d4-e155-4a60-8013-52811488053f None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:44.794990 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7232c6ed-41de-46cd-b80e-fec2df4613de None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:44.795157 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380e0a3d-83c5-4095-aeae-3b6caf348b5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:44.795157 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380e0a3d-83c5-4095-aeae-3b6caf348b5f None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:44.795157 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380e0a3d-83c5-4095-aeae-3b6caf348b5f None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:44.795157 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380e0a3d-83c5-4095-aeae-3b6caf348b5f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:44.795611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380e0a3d-83c5-4095-aeae-3b6caf348b5f None admin] RBAC: Adding query filter params (role.id=727f0fa598e54b9a929d2cf6277c875e, scope.project.id=65ee0f795c8b406ea7bc6f5ad7f3e336, user.id=68fae1e50b4347b692dc718993d6fe73) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:44.795611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380e0a3d-83c5-4095-aeae-3b6caf348b5f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:44.797661 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380e0a3d-83c5-4095-aeae-3b6caf348b5f None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:44.804189 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 544/1095] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:44:44 2022] GET /identity/v3/role_assignments?user.id=68fae1e50b4347b692dc718993d6fe73&scope.project.id=65ee0f795c8b406ea7bc6f5ad7f3e336&role.id=727f0fa598e54b9a929d2cf6277c875e => generated 608 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:44.818148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c627f91f-c49d-4b8e-bc27-660f80d42777 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:44.818789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c627f91f-c49d-4b8e-bc27-660f80d42777 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:44.819069 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c627f91f-c49d-4b8e-bc27-660f80d42777 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:44.819365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c627f91f-c49d-4b8e-bc27-660f80d42777 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:44.820213 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c627f91f-c49d-4b8e-bc27-660f80d42777 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:44.820590 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c627f91f-c49d-4b8e-bc27-660f80d42777 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:44.822097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c627f91f-c49d-4b8e-bc27-660f80d42777 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:44.831348 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 552/1096] 10.222.0.62 () {56 vars in 1115 bytes} [Fri Sep 23 13:44:44 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:44:44.837609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380e0a3d-83c5-4095-aeae-3b6caf348b5f None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:44.862621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:44.943288 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380e0a3d-83c5-4095-aeae-3b6caf348b5f None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:44.979287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692e273c-f771-477b-9433-62cb8ee94fd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:44.979937 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692e273c-f771-477b-9433-62cb8ee94fd4 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:44.980216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692e273c-f771-477b-9433-62cb8ee94fd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:44.980484 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692e273c-f771-477b-9433-62cb8ee94fd4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:44.993393 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692e273c-f771-477b-9433-62cb8ee94fd4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:44.994874 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692e273c-f771-477b-9433-62cb8ee94fd4 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:45.010476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-692e273c-f771-477b-9433-62cb8ee94fd4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:45.010476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:45.010476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:45.010476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:45.010476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:45.010476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:45.010476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:45.010476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:45.010476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:45.010476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:45.010476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:45.010476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:45.010476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:45.010476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:45.010476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:45.010476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:45.011721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:45.011721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:45.011721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:45.011721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:45.011721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:45.011721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:45.011721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:45.011721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:45.011721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:45.011721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:45.011721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:45.011721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:45.011721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:45.011721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:45.011721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:45.011721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:45.011721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:45.012695 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:45.012695 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:45.012695 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:45.012695 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 545/1097] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:45.018132 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c627f91f-c49d-4b8e-bc27-660f80d42777 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:45.081006 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:45.117813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c627f91f-c49d-4b8e-bc27-660f80d42777 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:45.151349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224a1025-9ea5-48fd-9a9f-291b7a849209 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:45.151945 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224a1025-9ea5-48fd-9a9f-291b7a849209 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:45.152219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224a1025-9ea5-48fd-9a9f-291b7a849209 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:45.152483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224a1025-9ea5-48fd-9a9f-291b7a849209 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:45.153115 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224a1025-9ea5-48fd-9a9f-291b7a849209 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:45.153414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224a1025-9ea5-48fd-9a9f-291b7a849209 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:45.154799 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224a1025-9ea5-48fd-9a9f-291b7a849209 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:45.166379 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 553/1098] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:45.172457 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692e273c-f771-477b-9433-62cb8ee94fd4 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:45.248931 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:45.278628 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692e273c-f771-477b-9433-62cb8ee94fd4 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:45.313364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaec3acb-538e-4f2e-875d-1087184535ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:45.314064 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaec3acb-538e-4f2e-875d-1087184535ae None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:45.314367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaec3acb-538e-4f2e-875d-1087184535ae None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:45.314644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaec3acb-538e-4f2e-875d-1087184535ae None admin] PATH_INFO: `/v3/users/nova` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:45.322542 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaec3acb-538e-4f2e-875d-1087184535ae None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:45.323969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaec3acb-538e-4f2e-875d-1087184535ae None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:45.335011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-aaec3acb-538e-4f2e-875d-1087184535ae None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 23 13:44:45.335011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:45.335011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:45.335011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:45.335011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:45.335011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:45.335011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:45.335011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:45.335011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:45.335011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:45.335011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:45.335011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:45.335011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:44:45.335011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:44:45.335011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:44:45.335011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:44:45.336020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:45.336020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:45.336020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:44:45.336020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:45.336020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:44:45.336020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:44:45.336020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:45.336020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:45.336020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:45.336020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:45.336020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:45.336020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:45.336020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:45.336020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:45.336020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:45.336020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:45.336020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:45.336020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:45.336958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:45.336958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:45.336958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:44:45.336958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:44:45.336958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:44:45.336958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:44:45.336958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:44:45.336958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:44:45.336958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 23 13:44:45.336958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:44:45.336958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 546/1099] 10.222.0.62 () {56 vars in 1093 bytes} [Fri Sep 23 13:44:45 2022] GET /identity/v3/users/nova => generated 82 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:45.341776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224a1025-9ea5-48fd-9a9f-291b7a849209 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:45.453507 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224a1025-9ea5-48fd-9a9f-291b7a849209 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:45.477208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:45.488150 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e307f545-aec3-4401-8ef9-ee0ddea65c7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:45.489012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e307f545-aec3-4401-8ef9-ee0ddea65c7c None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:45.489284 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e307f545-aec3-4401-8ef9-ee0ddea65c7c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:45.489545 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e307f545-aec3-4401-8ef9-ee0ddea65c7c None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:45.490255 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e307f545-aec3-4401-8ef9-ee0ddea65c7c None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:45.490621 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e307f545-aec3-4401-8ef9-ee0ddea65c7c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:45.492051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e307f545-aec3-4401-8ef9-ee0ddea65c7c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:45.509530 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 554/1100] 10.222.0.62 () {56 vars in 1133 bytes} [Fri Sep 23 13:44:45 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:45.515275 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaec3acb-538e-4f2e-875d-1087184535ae None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:45.619016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaec3acb-538e-4f2e-875d-1087184535ae None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:45.647562 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:45.652714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1810b4-5dae-4462-9312-975ed5e7c7eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:45.653322 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1810b4-5dae-4462-9312-975ed5e7c7eb None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:45.653603 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1810b4-5dae-4462-9312-975ed5e7c7eb None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:45.653876 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1810b4-5dae-4462-9312-975ed5e7c7eb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:45.730532 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d1810b4-5dae-4462-9312-975ed5e7c7eb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:45.731978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d1810b4-5dae-4462-9312-975ed5e7c7eb None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:45.836734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 547/1101] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:44:45 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 23 13:44:45.842565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e307f545-aec3-4401-8ef9-ee0ddea65c7c None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:45.947004 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e307f545-aec3-4401-8ef9-ee0ddea65c7c None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:45.981416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b2c292-db6c-4a50-a56d-09b8db263ad4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:45.982047 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b2c292-db6c-4a50-a56d-09b8db263ad4 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:45.982359 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b2c292-db6c-4a50-a56d-09b8db263ad4 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:45.982623 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b2c292-db6c-4a50-a56d-09b8db263ad4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:45.993263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b2c292-db6c-4a50-a56d-09b8db263ad4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:45.994732 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b2c292-db6c-4a50-a56d-09b8db263ad4 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:46.009616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-b0b2c292-db6c-4a50-a56d-09b8db263ad4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:46.009616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:46.009616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:46.009616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:46.009616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:46.009616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:46.009616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:46.009616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:46.009616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:46.009616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:46.009616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:46.009616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:46.009616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:44:46.009616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:44:46.009616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:44:46.009616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:44:46.010738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:46.010738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:46.010738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:46.010738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:46.010738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:46.010738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:46.010738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:46.010738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:46.010738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:46.010738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:46.010738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:46.010738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:46.010738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:46.010738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:46.010738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:46.010738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:46.010738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:44:46.011656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:44:46.011656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:44:46.011656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:46.011656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 555/1102] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:44:46.017009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1810b4-5dae-4462-9312-975ed5e7c7eb None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:46.048210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:46.126722 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1810b4-5dae-4462-9312-975ed5e7c7eb None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:46.163376 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21ac6d3-e1f0-44e9-b38a-82d13f3b96ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:46.163979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21ac6d3-e1f0-44e9-b38a-82d13f3b96ce None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:46.164251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21ac6d3-e1f0-44e9-b38a-82d13f3b96ce None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:46.164516 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21ac6d3-e1f0-44e9-b38a-82d13f3b96ce None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:46.165148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21ac6d3-e1f0-44e9-b38a-82d13f3b96ce None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:46.165445 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21ac6d3-e1f0-44e9-b38a-82d13f3b96ce None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:46.166839 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21ac6d3-e1f0-44e9-b38a-82d13f3b96ce None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:46.183612 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 548/1103] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:44:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:46.184934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b2c292-db6c-4a50-a56d-09b8db263ad4 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:46.226073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:46.294299 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b2c292-db6c-4a50-a56d-09b8db263ad4 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:46.331085 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f25266-253b-424d-b5d7-64108cb17f18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:46.331728 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f25266-253b-424d-b5d7-64108cb17f18 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:46.332012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f25266-253b-424d-b5d7-64108cb17f18 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:46.332284 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f25266-253b-424d-b5d7-64108cb17f18 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:46.338955 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f25266-253b-424d-b5d7-64108cb17f18 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:46.340391 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f25266-253b-424d-b5d7-64108cb17f18 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:46.351352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-e2f25266-253b-424d-b5d7-64108cb17f18 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:46.351352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:44:46.351352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:44:46.351352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:44:46.351352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:44:46.351352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:44:46.351352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:44:46.351352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:44:46.351352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:44:46.351352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:44:46.351352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:44:46.351352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:44:46.351352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:44:46.351352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:44:46.351352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:44:46.352410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:44:46.352410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:44:46.352410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:44:46.352410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:44:46.352410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:44:46.352410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:44:46.352410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:44:46.352410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:44:46.352410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:44:46.352410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:44:46.352410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:44:46.352410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:44:46.352410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:44:46.352410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:44:46.352410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:44:46.352410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:44:46.352410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:44:46.353514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:44:46.353514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:44:46.353514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:44:46.353514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:44:46.353514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:44:46.353514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:44:46.353514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:44:46.353514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:44:46.354476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 556/1104] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:44:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:44:46.358490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21ac6d3-e1f0-44e9-b38a-82d13f3b96ce None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:46.447665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:46.456607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21ac6d3-e1f0-44e9-b38a-82d13f3b96ce None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:46.489358 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81641b54-fc18-4549-8391-c5680e7eb952 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:46.489918 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81641b54-fc18-4549-8391-c5680e7eb952 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:46.490201 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81641b54-fc18-4549-8391-c5680e7eb952 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:46.490508 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81641b54-fc18-4549-8391-c5680e7eb952 None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:46.491199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81641b54-fc18-4549-8391-c5680e7eb952 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:46.491526 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81641b54-fc18-4549-8391-c5680e7eb952 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:46.492941 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81641b54-fc18-4549-8391-c5680e7eb952 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:46.505790 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 549/1105] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:44:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:46.511593 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f25266-253b-424d-b5d7-64108cb17f18 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:46.604088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:46.611227 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f25266-253b-424d-b5d7-64108cb17f18 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:46.647193 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f7f05c-2cd5-4990-b95b-a23118c8f5c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:46.647796 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f7f05c-2cd5-4990-b95b-a23118c8f5c9 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:46.648075 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f7f05c-2cd5-4990-b95b-a23118c8f5c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:46.648343 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f7f05c-2cd5-4990-b95b-a23118c8f5c9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:46.649062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f7f05c-2cd5-4990-b95b-a23118c8f5c9 None admin] RBAC: Adding query filter params (role.id=b2084c06d5a44297a9a7795d559fae94, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=f01ed94dbb384a178ad0a45a6f3d8c17) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:46.649365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f7f05c-2cd5-4990-b95b-a23118c8f5c9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:46.650821 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f7f05c-2cd5-4990-b95b-a23118c8f5c9 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:46.656633 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 557/1106] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:44:46 2022] GET /identity/v3/role_assignments?user.id=f01ed94dbb384a178ad0a45a6f3d8c17&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=b2084c06d5a44297a9a7795d559fae94 => generated 608 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:47.106995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ce8a9c-5b18-4bc3-b875-54a91ca49bc9 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:47.107545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ce8a9c-5b18-4bc3-b875-54a91ca49bc9 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:47.107545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ce8a9c-5b18-4bc3-b875-54a91ca49bc9 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:47.109468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 550/1107] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:47.115044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e769f0a-e11b-4da0-93fd-baae7ea37fda None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:47.115473 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e769f0a-e11b-4da0-93fd-baae7ea37fda None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:47.115582 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e769f0a-e11b-4da0-93fd-baae7ea37fda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:47.134940 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-6e769f0a-e11b-4da0-93fd-baae7ea37fda None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:47.136130 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-6e769f0a-e11b-4da0-93fd-baae7ea37fda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:47.211884 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 558/1108] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:47.486897 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ce8a9c-5b18-4bc3-b875-54a91ca49bc9 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:47.589275 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ce8a9c-5b18-4bc3-b875-54a91ca49bc9 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:47.622020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4998a459-fa6a-4bc0-9032-7209c47896ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:47.622656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4998a459-fa6a-4bc0-9032-7209c47896ac None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:47.622933 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4998a459-fa6a-4bc0-9032-7209c47896ac None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:47.623199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4998a459-fa6a-4bc0-9032-7209c47896ac None admin] PATH_INFO: `/v3/domains/default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:47.624995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4998a459-fa6a-4bc0-9032-7209c47896ac None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:47.626423 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4998a459-fa6a-4bc0-9032-7209c47896ac None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:47.627963 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 551/1109] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:44:47 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:47.633691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e769f0a-e11b-4da0-93fd-baae7ea37fda None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:47.715651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:47.727223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e769f0a-e11b-4da0-93fd-baae7ea37fda None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:47.760855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b607ae2d-9005-430d-8db9-1e9392b2f2a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:47.761375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b607ae2d-9005-430d-8db9-1e9392b2f2a2 None admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:47.761650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b607ae2d-9005-430d-8db9-1e9392b2f2a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:47.761914 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b607ae2d-9005-430d-8db9-1e9392b2f2a2 None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:47.762575 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b607ae2d-9005-430d-8db9-1e9392b2f2a2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:47.763980 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b607ae2d-9005-430d-8db9-1e9392b2f2a2 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:47.853954 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 559/1110] 10.222.0.62 () {60 vars in 1132 bytes} [Fri Sep 23 13:44:47 2022] POST /identity/v3/users => generated 286 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:44:50.029861 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4698d669-3b9c-4a84-924b-8785c293c20a None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:50.031090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4698d669-3b9c-4a84-924b-8785c293c20a None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:50.031090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4698d669-3b9c-4a84-924b-8785c293c20a None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:50.032434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 552/1111] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:50.037749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d698240-88ce-415b-8061-3a88bdee35ca None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:50.038120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d698240-88ce-415b-8061-3a88bdee35ca None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:50.038288 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d698240-88ce-415b-8061-3a88bdee35ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:50.056884 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-1d698240-88ce-415b-8061-3a88bdee35ca None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:50.058065 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-1d698240-88ce-415b-8061-3a88bdee35ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:50.122638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 560/1112] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:50.391329 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4698d669-3b9c-4a84-924b-8785c293c20a None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:50.412770 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:50.488491 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4698d669-3b9c-4a84-924b-8785c293c20a None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:50.521105 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0b2b97-8f95-473e-b789-d0ee98ec5469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:50.521716 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0b2b97-8f95-473e-b789-d0ee98ec5469 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:50.521992 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0b2b97-8f95-473e-b789-d0ee98ec5469 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:50.522259 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0b2b97-8f95-473e-b789-d0ee98ec5469 None admin] PATH_INFO: `/v3/roles/5ef6f90ed37e44adafc451b081a501d1` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:50.523632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0b2b97-8f95-473e-b789-d0ee98ec5469 None admin] RBAC: Authorizing `identity:get_role(role_id=5ef6f90ed37e44adafc451b081a501d1)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:50.524973 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab0b2b97-8f95-473e-b789-d0ee98ec5469 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:50.526265 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 553/1113] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:44:50 2022] GET /identity/v3/roles/5ef6f90ed37e44adafc451b081a501d1 => generated 223 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:50.532618 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d698240-88ce-415b-8061-3a88bdee35ca None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:50.603672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:50.627281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d698240-88ce-415b-8061-3a88bdee35ca None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:50.660753 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5540cd17-3ad7-4a98-a6eb-87d8ab6d7e7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:50.661310 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5540cd17-3ad7-4a98-a6eb-87d8ab6d7e7b None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:50.661587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5540cd17-3ad7-4a98-a6eb-87d8ab6d7e7b None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:50.661856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5540cd17-3ad7-4a98-a6eb-87d8ab6d7e7b None admin] PATH_INFO: `/v3/users/19457b17a1204f48ad712abd5856715b` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:50.675637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5540cd17-3ad7-4a98-a6eb-87d8ab6d7e7b None admin] RBAC: Authorizing `identity:get_user(user_id=19457b17a1204f48ad712abd5856715b)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:50.677062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5540cd17-3ad7-4a98-a6eb-87d8ab6d7e7b None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:50.678677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 561/1114] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:44:50 2022] GET /identity/v3/users/19457b17a1204f48ad712abd5856715b => generated 286 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:50.684351 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0b2b97-8f95-473e-b789-d0ee98ec5469 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:50.770438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:50.782195 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0b2b97-8f95-473e-b789-d0ee98ec5469 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:50.815537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd7edc3-5505-4a11-9a11-2107f4e4db2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:50.816124 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd7edc3-5505-4a11-9a11-2107f4e4db2c None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:50.816395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd7edc3-5505-4a11-9a11-2107f4e4db2c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:50.816658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd7edc3-5505-4a11-9a11-2107f4e4db2c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:50.881193 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd7edc3-5505-4a11-9a11-2107f4e4db2c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:50.882664 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd7edc3-5505-4a11-9a11-2107f4e4db2c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:50.963600 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:50.975950 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 554/1115] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:44:50 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 23 13:44:50.981776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5540cd17-3ad7-4a98-a6eb-87d8ab6d7e7b None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:51.076979 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5540cd17-3ad7-4a98-a6eb-87d8ab6d7e7b None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:51.105266 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:51.110067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daad4b42-73d2-4cd0-93b7-08b78aba9b2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:51.110712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daad4b42-73d2-4cd0-93b7-08b78aba9b2d None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:51.110989 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daad4b42-73d2-4cd0-93b7-08b78aba9b2d None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:51.111258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daad4b42-73d2-4cd0-93b7-08b78aba9b2d None admin] PATH_INFO: `/v3/projects/65ee0f795c8b406ea7bc6f5ad7f3e336` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:51.112363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daad4b42-73d2-4cd0-93b7-08b78aba9b2d None admin] RBAC: Authorizing `identity:get_project(project_id=65ee0f795c8b406ea7bc6f5ad7f3e336)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:51.114047 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daad4b42-73d2-4cd0-93b7-08b78aba9b2d None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:51.115686 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 562/1116] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:44:50 2022] GET /identity/v3/projects/65ee0f795c8b406ea7bc6f5ad7f3e336 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:51.121833 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd7edc3-5505-4a11-9a11-2107f4e4db2c None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:51.216426 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd7edc3-5505-4a11-9a11-2107f4e4db2c None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:51.248882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374ece54-321e-449d-83de-e1951bed451a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:51.249464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374ece54-321e-449d-83de-e1951bed451a None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:51.249737 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374ece54-321e-449d-83de-e1951bed451a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:51.250000 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374ece54-321e-449d-83de-e1951bed451a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:51.250734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374ece54-321e-449d-83de-e1951bed451a None admin] RBAC: Adding query filter params (role.id=5ef6f90ed37e44adafc451b081a501d1, scope.project.id=65ee0f795c8b406ea7bc6f5ad7f3e336, user.id=19457b17a1204f48ad712abd5856715b) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:51.251035 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374ece54-321e-449d-83de-e1951bed451a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:51.252425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374ece54-321e-449d-83de-e1951bed451a None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:51.257551 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 555/1117] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:44:51 2022] GET /identity/v3/role_assignments?user.id=19457b17a1204f48ad712abd5856715b&scope.project.id=65ee0f795c8b406ea7bc6f5ad7f3e336&role.id=5ef6f90ed37e44adafc451b081a501d1 => generated 259 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:53.351859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b79956-2dc0-49b8-9347-44a52b6e904d None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:53.352399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b79956-2dc0-49b8-9347-44a52b6e904d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:53.352399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b79956-2dc0-49b8-9347-44a52b6e904d None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:53.353727 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 563/1118] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:53.359649 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a53c1f0-b74c-4ff4-9ce0-07d3462af728 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:53.360034 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a53c1f0-b74c-4ff4-9ce0-07d3462af728 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:53.360190 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a53c1f0-b74c-4ff4-9ce0-07d3462af728 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:53.379093 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-5a53c1f0-b74c-4ff4-9ce0-07d3462af728 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:53.380275 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-5a53c1f0-b74c-4ff4-9ce0-07d3462af728 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:53.425717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:53.445315 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 556/1119] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:53 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:53.717492 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b79956-2dc0-49b8-9347-44a52b6e904d None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:53.816098 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b79956-2dc0-49b8-9347-44a52b6e904d None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:53.851105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9462fe47-5511-43e6-a8e6-8865c264b30e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:53.851718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9462fe47-5511-43e6-a8e6-8865c264b30e None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:53.851993 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9462fe47-5511-43e6-a8e6-8865c264b30e None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:53.852261 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9462fe47-5511-43e6-a8e6-8865c264b30e None admin] PATH_INFO: `/v3/roles/5ef6f90ed37e44adafc451b081a501d1` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:53.853629 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9462fe47-5511-43e6-a8e6-8865c264b30e None admin] RBAC: Authorizing `identity:get_role(role_id=5ef6f90ed37e44adafc451b081a501d1)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:53.855000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9462fe47-5511-43e6-a8e6-8865c264b30e None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:53.856295 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 564/1120] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:44:53 2022] GET /identity/v3/roles/5ef6f90ed37e44adafc451b081a501d1 => generated 223 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:53.863208 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a53c1f0-b74c-4ff4-9ce0-07d3462af728 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:53.955959 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a53c1f0-b74c-4ff4-9ce0-07d3462af728 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:53.989632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de25063f-24f3-47e4-bd0a-3b177870b223 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:53.990194 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de25063f-24f3-47e4-bd0a-3b177870b223 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:53.990509 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de25063f-24f3-47e4-bd0a-3b177870b223 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:53.990780 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de25063f-24f3-47e4-bd0a-3b177870b223 None admin] PATH_INFO: `/v3/users/19457b17a1204f48ad712abd5856715b` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:53.991888 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de25063f-24f3-47e4-bd0a-3b177870b223 None admin] RBAC: Authorizing `identity:get_user(user_id=19457b17a1204f48ad712abd5856715b)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:53.993293 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de25063f-24f3-47e4-bd0a-3b177870b223 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:53.994877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 557/1121] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:44:53 2022] GET /identity/v3/users/19457b17a1204f48ad712abd5856715b => generated 286 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:54.000613 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9462fe47-5511-43e6-a8e6-8865c264b30e None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:54.066659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:54.100037 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9462fe47-5511-43e6-a8e6-8865c264b30e None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:54.133692 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40557c53-f6e0-43fb-aaea-33a3eba4b02d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:54.134306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40557c53-f6e0-43fb-aaea-33a3eba4b02d None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:54.134623 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40557c53-f6e0-43fb-aaea-33a3eba4b02d None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:54.134894 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40557c53-f6e0-43fb-aaea-33a3eba4b02d None admin] PATH_INFO: `/v3/projects/65ee0f795c8b406ea7bc6f5ad7f3e336` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:54.136002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40557c53-f6e0-43fb-aaea-33a3eba4b02d None admin] RBAC: Authorizing `identity:get_project(project_id=65ee0f795c8b406ea7bc6f5ad7f3e336)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:54.137417 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40557c53-f6e0-43fb-aaea-33a3eba4b02d None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:54.139579 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 565/1122] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:44:53 2022] GET /identity/v3/projects/65ee0f795c8b406ea7bc6f5ad7f3e336 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:54.145825 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de25063f-24f3-47e4-bd0a-3b177870b223 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:54.195773 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:54.246074 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de25063f-24f3-47e4-bd0a-3b177870b223 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:54.280304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91968c7a-7b8c-4e82-a048-e594726334ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:54.280941 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91968c7a-7b8c-4e82-a048-e594726334ea None admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:54.281215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91968c7a-7b8c-4e82-a048-e594726334ea None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:54.281480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91968c7a-7b8c-4e82-a048-e594726334ea None admin] PATH_INFO: `/v3/projects/65ee0f795c8b406ea7bc6f5ad7f3e336/users/19457b17a1204f48ad712abd5856715b/roles/5ef6f90ed37e44adafc451b081a501d1` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:54.283621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91968c7a-7b8c-4e82-a048-e594726334ea None admin] RBAC: Authorizing `identity:create_grant(project_id=65ee0f795c8b406ea7bc6f5ad7f3e336, user_id=19457b17a1204f48ad712abd5856715b, role_id=5ef6f90ed37e44adafc451b081a501d1)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:54.285030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91968c7a-7b8c-4e82-a048-e594726334ea None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:54.386851 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 558/1123] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:44:54 2022] PUT /identity/v3/projects/65ee0f795c8b406ea7bc6f5ad7f3e336/users/19457b17a1204f48ad712abd5856715b/roles/5ef6f90ed37e44adafc451b081a501d1 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:44:56.517723 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29ec6ab-7a04-4a10-b462-b36186f6347f None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:56.518233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29ec6ab-7a04-4a10-b462-b36186f6347f None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:56.518233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29ec6ab-7a04-4a10-b462-b36186f6347f None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:56.519576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 566/1124] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:56.524698 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ca2eb9-a290-40a5-a837-c9103521cae4 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:56.525107 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ca2eb9-a290-40a5-a837-c9103521cae4 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:56.525245 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ca2eb9-a290-40a5-a837-c9103521cae4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:56.543092 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-06ca2eb9-a290-40a5-a837-c9103521cae4 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:56.544280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-06ca2eb9-a290-40a5-a837-c9103521cae4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:56.609902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 559/1125] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:56 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:44:56.879767 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29ec6ab-7a04-4a10-b462-b36186f6347f None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:56.977128 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29ec6ab-7a04-4a10-b462-b36186f6347f None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:57.009811 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c293fbc2-3de0-4293-a28b-1fb20f4a4084 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:57.010439 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c293fbc2-3de0-4293-a28b-1fb20f4a4084 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:57.010715 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c293fbc2-3de0-4293-a28b-1fb20f4a4084 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:57.010980 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c293fbc2-3de0-4293-a28b-1fb20f4a4084 None admin] PATH_INFO: `/v3/roles/5ef6f90ed37e44adafc451b081a501d1` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:57.012333 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c293fbc2-3de0-4293-a28b-1fb20f4a4084 None admin] RBAC: Authorizing `identity:get_role(role_id=5ef6f90ed37e44adafc451b081a501d1)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:57.013794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c293fbc2-3de0-4293-a28b-1fb20f4a4084 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:57.015124 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 567/1126] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:44:56 2022] GET /identity/v3/roles/5ef6f90ed37e44adafc451b081a501d1 => generated 223 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:57.020832 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ca2eb9-a290-40a5-a837-c9103521cae4 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:57.061098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:57.116403 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ca2eb9-a290-40a5-a837-c9103521cae4 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:57.148961 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2122ead8-7dfa-4c68-96e9-0183e87593f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:57.149513 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2122ead8-7dfa-4c68-96e9-0183e87593f4 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:57.149787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2122ead8-7dfa-4c68-96e9-0183e87593f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:57.150061 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2122ead8-7dfa-4c68-96e9-0183e87593f4 None admin] PATH_INFO: `/v3/users/19457b17a1204f48ad712abd5856715b` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:57.151187 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2122ead8-7dfa-4c68-96e9-0183e87593f4 None admin] RBAC: Authorizing `identity:get_user(user_id=19457b17a1204f48ad712abd5856715b)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:57.152601 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2122ead8-7dfa-4c68-96e9-0183e87593f4 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:57.154837 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 560/1127] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:44:57 2022] GET /identity/v3/users/19457b17a1204f48ad712abd5856715b => generated 286 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:57.160445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c293fbc2-3de0-4293-a28b-1fb20f4a4084 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:57.164236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:57.256349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c293fbc2-3de0-4293-a28b-1fb20f4a4084 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:57.288998 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8741d90-2c42-44d4-a50b-e4bcddd46dd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:57.289588 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8741d90-2c42-44d4-a50b-e4bcddd46dd5 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:57.289863 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8741d90-2c42-44d4-a50b-e4bcddd46dd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:57.290129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8741d90-2c42-44d4-a50b-e4bcddd46dd5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:57.354378 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8741d90-2c42-44d4-a50b-e4bcddd46dd5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:57.355811 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8741d90-2c42-44d4-a50b-e4bcddd46dd5 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:57.358345 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:57.449294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 568/1128] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:44:57 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 23 13:44:57.456093 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2122ead8-7dfa-4c68-96e9-0183e87593f4 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:57.552233 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2122ead8-7dfa-4c68-96e9-0183e87593f4 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:57.561564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:57.585450 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aca471d-4353-4b04-bba5-a24bcea2c539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:57.586061 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aca471d-4353-4b04-bba5-a24bcea2c539 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:57.586360 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aca471d-4353-4b04-bba5-a24bcea2c539 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:57.586645 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aca471d-4353-4b04-bba5-a24bcea2c539 None admin] PATH_INFO: `/v3/projects/65ee0f795c8b406ea7bc6f5ad7f3e336` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:57.587749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aca471d-4353-4b04-bba5-a24bcea2c539 None admin] RBAC: Authorizing `identity:get_project(project_id=65ee0f795c8b406ea7bc6f5ad7f3e336)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:57.589157 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aca471d-4353-4b04-bba5-a24bcea2c539 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:57.590799 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 561/1129] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:44:57 2022] GET /identity/v3/projects/65ee0f795c8b406ea7bc6f5ad7f3e336 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:57.597057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8741d90-2c42-44d4-a50b-e4bcddd46dd5 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:44:57.692097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8741d90-2c42-44d4-a50b-e4bcddd46dd5 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:44:57.694346 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:44:57.724614 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73df7a87-dba1-4852-bc7c-ba6107fe1322 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:44:57.725196 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73df7a87-dba1-4852-bc7c-ba6107fe1322 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:57.725470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73df7a87-dba1-4852-bc7c-ba6107fe1322 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:57.725735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73df7a87-dba1-4852-bc7c-ba6107fe1322 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:57.726471 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73df7a87-dba1-4852-bc7c-ba6107fe1322 None admin] RBAC: Adding query filter params (role.id=5ef6f90ed37e44adafc451b081a501d1, scope.project.id=65ee0f795c8b406ea7bc6f5ad7f3e336, user.id=19457b17a1204f48ad712abd5856715b) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:44:57.726774 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73df7a87-dba1-4852-bc7c-ba6107fe1322 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:44:57.728176 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73df7a87-dba1-4852-bc7c-ba6107fe1322 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:44:57.733721 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 569/1130] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:44:57 2022] GET /identity/v3/role_assignments?user.id=19457b17a1204f48ad712abd5856715b&scope.project.id=65ee0f795c8b406ea7bc6f5ad7f3e336&role.id=5ef6f90ed37e44adafc451b081a501d1 => generated 608 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:44:59.902360 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77af57a-8428-44f5-8359-a70691f5502f None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:59.902923 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77af57a-8428-44f5-8359-a70691f5502f None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:59.902923 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77af57a-8428-44f5-8359-a70691f5502f None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:59.904809 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 562/1131] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:44:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:44:59.910097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b921cd95-2369-4d8f-86e2-ba00cbba92c1 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:44:59.910493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b921cd95-2369-4d8f-86e2-ba00cbba92c1 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:44:59.910683 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b921cd95-2369-4d8f-86e2-ba00cbba92c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:44:59.929762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-b921cd95-2369-4d8f-86e2-ba00cbba92c1 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:44:59.931000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-b921cd95-2369-4d8f-86e2-ba00cbba92c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:44:59.995343 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 570/1132] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:44:59 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:45:00.263183 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77af57a-8428-44f5-8359-a70691f5502f None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:00.363371 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77af57a-8428-44f5-8359-a70691f5502f None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:00.396026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56a07e3-f153-4782-a7dd-6fd317de43fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:00.396635 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56a07e3-f153-4782-a7dd-6fd317de43fe None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:00.396909 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56a07e3-f153-4782-a7dd-6fd317de43fe None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:00.397174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56a07e3-f153-4782-a7dd-6fd317de43fe None admin] PATH_INFO: `/v3/domains/default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:00.398274 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56a07e3-f153-4782-a7dd-6fd317de43fe None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:00.399700 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56a07e3-f153-4782-a7dd-6fd317de43fe None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:00.401239 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 563/1133] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:45:00 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:00.407633 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b921cd95-2369-4d8f-86e2-ba00cbba92c1 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:00.504176 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:00.507866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b921cd95-2369-4d8f-86e2-ba00cbba92c1 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:00.542233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827d2527-7980-476e-b0a5-910087114269 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:00.542864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827d2527-7980-476e-b0a5-910087114269 None admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:00.543144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827d2527-7980-476e-b0a5-910087114269 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:00.543416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827d2527-7980-476e-b0a5-910087114269 None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:00.544061 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827d2527-7980-476e-b0a5-910087114269 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:00.546662 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827d2527-7980-476e-b0a5-910087114269 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:00.639233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 571/1134] 10.222.0.62 () {60 vars in 1141 bytes} [Fri Sep 23 13:45:00 2022] POST /identity/v3/projects => generated 311 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:45:02.804139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727a4ecb-ad93-4374-95cb-fac4f8615c46 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:02.804668 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727a4ecb-ad93-4374-95cb-fac4f8615c46 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:02.804668 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727a4ecb-ad93-4374-95cb-fac4f8615c46 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:02.806020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 564/1135] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:45:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:45:02.811766 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92193568-215d-4212-9794-518cab739086 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:02.812176 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92193568-215d-4212-9794-518cab739086 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:02.812311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92193568-215d-4212-9794-518cab739086 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:02.830629 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-92193568-215d-4212-9794-518cab739086 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:45:02.831804 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-92193568-215d-4212-9794-518cab739086 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:45:02.895979 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 572/1136] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:45:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 23 13:45:03.167923 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727a4ecb-ad93-4374-95cb-fac4f8615c46 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:03.214468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:03.267614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727a4ecb-ad93-4374-95cb-fac4f8615c46 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:03.299739 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404302b0-7d17-4fac-bd43-5bdfb8de2eb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:03.300351 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404302b0-7d17-4fac-bd43-5bdfb8de2eb2 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:03.300625 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404302b0-7d17-4fac-bd43-5bdfb8de2eb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:03.300889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404302b0-7d17-4fac-bd43-5bdfb8de2eb2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:03.301977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404302b0-7d17-4fac-bd43-5bdfb8de2eb2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:03.303404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404302b0-7d17-4fac-bd43-5bdfb8de2eb2 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:03.304946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 565/1137] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:45:03 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:03.310815 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92193568-215d-4212-9794-518cab739086 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:03.387661 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:03.406867 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92193568-215d-4212-9794-518cab739086 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:03.439113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3a5cc8-6c19-4580-b6e6-61df2b6eec1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:03.439651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3a5cc8-6c19-4580-b6e6-61df2b6eec1c None admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:03.439928 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3a5cc8-6c19-4580-b6e6-61df2b6eec1c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:03.440210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3a5cc8-6c19-4580-b6e6-61df2b6eec1c None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:03.441047 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb3a5cc8-6c19-4580-b6e6-61df2b6eec1c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:03.443122 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb3a5cc8-6c19-4580-b6e6-61df2b6eec1c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:03.515871 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 573/1138] 10.222.0.62 () {60 vars in 1132 bytes} [Fri Sep 23 13:45:03 2022] POST /identity/v3/users => generated 286 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:45:05.708765 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52c7966-c0fb-47ae-b8d7-85435190e9cb None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:05.709289 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52c7966-c0fb-47ae-b8d7-85435190e9cb None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:05.709383 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52c7966-c0fb-47ae-b8d7-85435190e9cb None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:05.710695 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 566/1139] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:45:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:45:05.716146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059f7acb-f314-4f44-8f14-d6f0170c8136 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:05.716658 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059f7acb-f314-4f44-8f14-d6f0170c8136 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:05.716742 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059f7acb-f314-4f44-8f14-d6f0170c8136 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:05.735012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-059f7acb-f314-4f44-8f14-d6f0170c8136 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:45:05.736196 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-059f7acb-f314-4f44-8f14-d6f0170c8136 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:45:05.801769 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 574/1140] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:45:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:45:06.076915 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52c7966-c0fb-47ae-b8d7-85435190e9cb None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:06.174939 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52c7966-c0fb-47ae-b8d7-85435190e9cb None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:06.187563 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:06.207976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6235c379-018c-49be-a008-6e0a9f915659 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:06.208621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6235c379-018c-49be-a008-6e0a9f915659 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:06.208896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6235c379-018c-49be-a008-6e0a9f915659 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:06.209160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6235c379-018c-49be-a008-6e0a9f915659 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:06.220466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6235c379-018c-49be-a008-6e0a9f915659 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:45:06.225242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-6235c379-018c-49be-a008-6e0a9f915659 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:45:06.225242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:45:06.225242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:45:06.225242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:45:06.225242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:45:06.225242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:45:06.225242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:45:06.225242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:45:06.225242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:45:06.225242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:45:06.225242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:45:06.225242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:45:06.225242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:45:06.225242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:45:06.225242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:45:06.225242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:45:06.226359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:45:06.226359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:45:06.226359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:45:06.226359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:45:06.226359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:45:06.226359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:45:06.226359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:45:06.226359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:45:06.226359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:45:06.226359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:45:06.226359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:45:06.226359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:45:06.226359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:45:06.226359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:45:06.226359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:45:06.226359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:45:06.227414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:45:06.227414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:45:06.227414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:45:06.227414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:45:06.227414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:45:06.227414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:45:06.227414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:45:06.227414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:45:06.227414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:45:06.227414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:45:06.227414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:45:06.227414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:45:06.227414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 567/1141] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:45:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:45:06.232578 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059f7acb-f314-4f44-8f14-d6f0170c8136 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:06.300386 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:06.334026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059f7acb-f314-4f44-8f14-d6f0170c8136 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:06.368051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1bb1ab-9616-43c8-9780-bb58546242dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:06.368649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1bb1ab-9616-43c8-9780-bb58546242dd None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:06.368928 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1bb1ab-9616-43c8-9780-bb58546242dd None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:06.369195 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1bb1ab-9616-43c8-9780-bb58546242dd None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:06.369851 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1bb1ab-9616-43c8-9780-bb58546242dd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:45:06.370150 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1bb1ab-9616-43c8-9780-bb58546242dd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:06.371540 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d1bb1ab-9616-43c8-9780-bb58546242dd None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:06.379752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 575/1142] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:45:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:06.385754 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6235c379-018c-49be-a008-6e0a9f915659 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:06.484315 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6235c379-018c-49be-a008-6e0a9f915659 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:06.517160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af58813b-7bef-44ef-a131-ff9f8dacc45a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:06.517766 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af58813b-7bef-44ef-a131-ff9f8dacc45a None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:06.518040 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af58813b-7bef-44ef-a131-ff9f8dacc45a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:06.518305 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af58813b-7bef-44ef-a131-ff9f8dacc45a None admin] PATH_INFO: `/v3/users/f5d137418041465b9ad4de99e2059e2a` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:06.530757 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af58813b-7bef-44ef-a131-ff9f8dacc45a None admin] RBAC: Authorizing `identity:get_user(user_id=f5d137418041465b9ad4de99e2059e2a)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:06.532190 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af58813b-7bef-44ef-a131-ff9f8dacc45a None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:06.533744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 568/1143] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:45:06 2022] GET /identity/v3/users/f5d137418041465b9ad4de99e2059e2a => generated 286 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:06.539537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1bb1ab-9616-43c8-9780-bb58546242dd None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:06.638525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d1bb1ab-9616-43c8-9780-bb58546242dd None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:06.663272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:06.670771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f4c357-d2be-47b7-894a-aaef526d4f61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:06.671366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f4c357-d2be-47b7-894a-aaef526d4f61 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:06.671645 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f4c357-d2be-47b7-894a-aaef526d4f61 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:06.671914 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f4c357-d2be-47b7-894a-aaef526d4f61 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:06.737230 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f4c357-d2be-47b7-894a-aaef526d4f61 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:06.738679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f4c357-d2be-47b7-894a-aaef526d4f61 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:06.833258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 576/1144] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:45:06 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 23 13:45:06.839337 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af58813b-7bef-44ef-a131-ff9f8dacc45a None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:06.868864 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:06.936311 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af58813b-7bef-44ef-a131-ff9f8dacc45a None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:06.968265 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc96f0d2-68a7-4adc-8c22-80baab6e0eb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:06.968878 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc96f0d2-68a7-4adc-8c22-80baab6e0eb6 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:06.969156 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc96f0d2-68a7-4adc-8c22-80baab6e0eb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:06.969426 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc96f0d2-68a7-4adc-8c22-80baab6e0eb6 None admin] PATH_INFO: `/v3/projects/30c69f3ee4ff4852b4e363b0d1b33fdb` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:06.970548 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc96f0d2-68a7-4adc-8c22-80baab6e0eb6 None admin] RBAC: Authorizing `identity:get_project(project_id=30c69f3ee4ff4852b4e363b0d1b33fdb)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:06.971953 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc96f0d2-68a7-4adc-8c22-80baab6e0eb6 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:06.973552 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 569/1145] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:45:06 2022] GET /identity/v3/projects/30c69f3ee4ff4852b4e363b0d1b33fdb => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:06.979701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f4c357-d2be-47b7-894a-aaef526d4f61 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:07.007187 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:07.075588 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f4c357-d2be-47b7-894a-aaef526d4f61 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:07.107879 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b237a4-6deb-461c-8238-8ffaab749aea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:07.108463 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b237a4-6deb-461c-8238-8ffaab749aea None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:07.108734 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b237a4-6deb-461c-8238-8ffaab749aea None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:07.108998 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b237a4-6deb-461c-8238-8ffaab749aea None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:07.109704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b237a4-6deb-461c-8238-8ffaab749aea None admin] RBAC: Adding query filter params (role.id=727f0fa598e54b9a929d2cf6277c875e, scope.project.id=30c69f3ee4ff4852b4e363b0d1b33fdb, user.id=f5d137418041465b9ad4de99e2059e2a) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:45:07.110005 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b237a4-6deb-461c-8238-8ffaab749aea None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:07.111709 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b237a4-6deb-461c-8238-8ffaab749aea None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:07.116826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 577/1146] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:45:06 2022] GET /identity/v3/role_assignments?user.id=f5d137418041465b9ad4de99e2059e2a&scope.project.id=30c69f3ee4ff4852b4e363b0d1b33fdb&role.id=727f0fa598e54b9a929d2cf6277c875e => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:09.223035 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13072af8-81e8-48e3-b294-226ecf66b3f3 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:09.223495 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13072af8-81e8-48e3-b294-226ecf66b3f3 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:09.223586 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13072af8-81e8-48e3-b294-226ecf66b3f3 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:09.224983 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 570/1147] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:45:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:45:09.230083 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb61ddaa-ecbc-44bc-b35f-e8d9cad203d4 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:09.230513 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb61ddaa-ecbc-44bc-b35f-e8d9cad203d4 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:09.230660 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb61ddaa-ecbc-44bc-b35f-e8d9cad203d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:09.249818 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-cb61ddaa-ecbc-44bc-b35f-e8d9cad203d4 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:45:09.251017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-cb61ddaa-ecbc-44bc-b35f-e8d9cad203d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:45:09.317138 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 578/1148] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:45:09 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:45:09.583292 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13072af8-81e8-48e3-b294-226ecf66b3f3 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:09.678455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:09.682963 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13072af8-81e8-48e3-b294-226ecf66b3f3 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:09.714750 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1229d90a-7caf-42a4-8db7-54cb6cfba357 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:09.715942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1229d90a-7caf-42a4-8db7-54cb6cfba357 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:09.716381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1229d90a-7caf-42a4-8db7-54cb6cfba357 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:09.716651 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1229d90a-7caf-42a4-8db7-54cb6cfba357 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:09.726579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1229d90a-7caf-42a4-8db7-54cb6cfba357 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:45:09.732046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-1229d90a-7caf-42a4-8db7-54cb6cfba357 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:45:09.732046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:45:09.732046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:45:09.732046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:45:09.732046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:45:09.732046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:45:09.732046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:45:09.732046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:45:09.732046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:45:09.732046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:45:09.732046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:45:09.732046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:45:09.732046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:45:09.732046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:45:09.732046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:45:09.732046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:45:09.733140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:45:09.733140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:45:09.733140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:45:09.733140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:45:09.733140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:45:09.733140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:45:09.733140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:45:09.733140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:45:09.733140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:45:09.733140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:45:09.733140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:45:09.733140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:45:09.733140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:45:09.733140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:45:09.733140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:45:09.733140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:45:09.734182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:45:09.734182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:45:09.734182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:45:09.734182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:45:09.734182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:45:09.734182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:45:09.734182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:45:09.734182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:45:09.734182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:45:09.734182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:45:09.734182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:45:09.734182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:45:09.734182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 571/1149] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:45:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:45:09.739453 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb61ddaa-ecbc-44bc-b35f-e8d9cad203d4 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:09.763423 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:09.835697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb61ddaa-ecbc-44bc-b35f-e8d9cad203d4 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:09.868552 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70f459c-7d69-4388-a5ed-cb6d41120801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:09.869144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70f459c-7d69-4388-a5ed-cb6d41120801 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:09.869425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70f459c-7d69-4388-a5ed-cb6d41120801 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:09.869696 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70f459c-7d69-4388-a5ed-cb6d41120801 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:09.870370 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70f459c-7d69-4388-a5ed-cb6d41120801 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:45:09.870674 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70f459c-7d69-4388-a5ed-cb6d41120801 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:09.872744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70f459c-7d69-4388-a5ed-cb6d41120801 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:09.880469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 579/1150] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:45:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:09.886523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1229d90a-7caf-42a4-8db7-54cb6cfba357 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:09.982710 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1229d90a-7caf-42a4-8db7-54cb6cfba357 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:10.014961 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ce598f-e2f2-4cd5-9610-b89671aff223 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:10.015565 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ce598f-e2f2-4cd5-9610-b89671aff223 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:10.015840 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ce598f-e2f2-4cd5-9610-b89671aff223 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:10.016108 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ce598f-e2f2-4cd5-9610-b89671aff223 None admin] PATH_INFO: `/v3/users/f5d137418041465b9ad4de99e2059e2a` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:10.017201 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ce598f-e2f2-4cd5-9610-b89671aff223 None admin] RBAC: Authorizing `identity:get_user(user_id=f5d137418041465b9ad4de99e2059e2a)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:10.018629 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ce598f-e2f2-4cd5-9610-b89671aff223 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:10.020172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 572/1151] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:45:09 2022] GET /identity/v3/users/f5d137418041465b9ad4de99e2059e2a => generated 286 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:10.025889 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70f459c-7d69-4388-a5ed-cb6d41120801 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:10.108846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:10.124998 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70f459c-7d69-4388-a5ed-cb6d41120801 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:10.157302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64ef4f0-719a-41ab-9efb-ac2ed1c58f64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:10.157952 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64ef4f0-719a-41ab-9efb-ac2ed1c58f64 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:10.158231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64ef4f0-719a-41ab-9efb-ac2ed1c58f64 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:10.158529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64ef4f0-719a-41ab-9efb-ac2ed1c58f64 None admin] PATH_INFO: `/v3/projects/30c69f3ee4ff4852b4e363b0d1b33fdb` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:10.159641 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c64ef4f0-719a-41ab-9efb-ac2ed1c58f64 None admin] RBAC: Authorizing `identity:get_project(project_id=30c69f3ee4ff4852b4e363b0d1b33fdb)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:10.161055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c64ef4f0-719a-41ab-9efb-ac2ed1c58f64 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:10.162714 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 580/1152] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:45:10 2022] GET /identity/v3/projects/30c69f3ee4ff4852b4e363b0d1b33fdb => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:10.169201 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ce598f-e2f2-4cd5-9610-b89671aff223 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:10.197345 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:10.269721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ce598f-e2f2-4cd5-9610-b89671aff223 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:10.304254 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2752eb14-1708-40bb-8b27-ae804832c33c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:10.304899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2752eb14-1708-40bb-8b27-ae804832c33c None admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:10.305182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2752eb14-1708-40bb-8b27-ae804832c33c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:10.305455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2752eb14-1708-40bb-8b27-ae804832c33c None admin] PATH_INFO: `/v3/projects/30c69f3ee4ff4852b4e363b0d1b33fdb/users/f5d137418041465b9ad4de99e2059e2a/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:10.307368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2752eb14-1708-40bb-8b27-ae804832c33c None admin] RBAC: Authorizing `identity:create_grant(project_id=30c69f3ee4ff4852b4e363b0d1b33fdb, user_id=f5d137418041465b9ad4de99e2059e2a, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:10.308790 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2752eb14-1708-40bb-8b27-ae804832c33c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:10.391274 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 573/1153] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:45:10 2022] PUT /identity/v3/projects/30c69f3ee4ff4852b4e363b0d1b33fdb/users/f5d137418041465b9ad4de99e2059e2a/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:45:12.503068 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad7eb72-76f6-49e4-8a8c-e34043f70387 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:12.503588 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad7eb72-76f6-49e4-8a8c-e34043f70387 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:12.503588 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad7eb72-76f6-49e4-8a8c-e34043f70387 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:12.504856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 581/1154] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:45:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:45:12.510027 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc006f10-5df3-4c76-9bc3-fffd898e8301 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:12.510485 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc006f10-5df3-4c76-9bc3-fffd898e8301 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:12.510601 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc006f10-5df3-4c76-9bc3-fffd898e8301 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:12.529046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-dc006f10-5df3-4c76-9bc3-fffd898e8301 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:45:12.530210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-dc006f10-5df3-4c76-9bc3-fffd898e8301 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:45:12.594453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 574/1155] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:45:12 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:45:12.865484 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad7eb72-76f6-49e4-8a8c-e34043f70387 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:12.964010 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad7eb72-76f6-49e4-8a8c-e34043f70387 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:12.997860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57ed0c9-354d-40fe-8788-629652326b12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:12.998495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57ed0c9-354d-40fe-8788-629652326b12 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:12.998774 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57ed0c9-354d-40fe-8788-629652326b12 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:12.999042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57ed0c9-354d-40fe-8788-629652326b12 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:13.008727 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57ed0c9-354d-40fe-8788-629652326b12 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:45:13.013520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-f57ed0c9-354d-40fe-8788-629652326b12 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:45:13.013520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:45:13.013520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:45:13.013520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:45:13.013520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:45:13.013520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:45:13.013520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:45:13.013520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:45:13.013520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:45:13.013520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:45:13.013520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:45:13.013520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:45:13.013520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:45:13.013520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:45:13.013520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:45:13.013520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:45:13.014636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:45:13.014636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:45:13.014636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:45:13.014636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:45:13.014636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:45:13.014636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:45:13.014636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:45:13.014636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:45:13.014636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:45:13.014636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:45:13.014636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:45:13.014636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:45:13.014636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:45:13.014636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:45:13.014636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:45:13.014636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:45:13.015797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:45:13.015797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:45:13.015797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:45:13.015797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:45:13.015797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:45:13.015797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:45:13.015797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:45:13.015797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:45:13.015797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:45:13.015797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:45:13.015797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:45:13.015797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:45:13.015797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 582/1156] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:45:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:45:13.020487 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc006f10-5df3-4c76-9bc3-fffd898e8301 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:13.037250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:13.115656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc006f10-5df3-4c76-9bc3-fffd898e8301 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:13.148096 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d802cb1-1554-415e-9f47-4ed6e06a45fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:13.148685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d802cb1-1554-415e-9f47-4ed6e06a45fc None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:13.148963 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d802cb1-1554-415e-9f47-4ed6e06a45fc None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:13.149229 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d802cb1-1554-415e-9f47-4ed6e06a45fc None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:13.149878 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d802cb1-1554-415e-9f47-4ed6e06a45fc None admin] RBAC: Adding query filter params (name=admin) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:45:13.150173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d802cb1-1554-415e-9f47-4ed6e06a45fc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:13.151559 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d802cb1-1554-415e-9f47-4ed6e06a45fc None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:13.159383 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 575/1157] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:45:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:13.165271 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57ed0c9-354d-40fe-8788-629652326b12 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:13.180711 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:13.264930 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57ed0c9-354d-40fe-8788-629652326b12 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:13.298960 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12ec981-f15a-49b6-85f9-35b17645f1cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:13.299574 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12ec981-f15a-49b6-85f9-35b17645f1cf None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:13.299851 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12ec981-f15a-49b6-85f9-35b17645f1cf None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:13.300116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12ec981-f15a-49b6-85f9-35b17645f1cf None admin] PATH_INFO: `/v3/users/f5d137418041465b9ad4de99e2059e2a` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:13.301216 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12ec981-f15a-49b6-85f9-35b17645f1cf None admin] RBAC: Authorizing `identity:get_user(user_id=f5d137418041465b9ad4de99e2059e2a)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:13.302656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12ec981-f15a-49b6-85f9-35b17645f1cf None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:13.304217 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 583/1158] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:45:13 2022] GET /identity/v3/users/f5d137418041465b9ad4de99e2059e2a => generated 286 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:13.310059 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d802cb1-1554-415e-9f47-4ed6e06a45fc None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:13.385333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:13.407080 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d802cb1-1554-415e-9f47-4ed6e06a45fc None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:13.440268 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d99e44-a91b-46ad-91a1-98aa4f0c252a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:13.440864 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d99e44-a91b-46ad-91a1-98aa4f0c252a None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:13.441144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d99e44-a91b-46ad-91a1-98aa4f0c252a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:13.441414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d99e44-a91b-46ad-91a1-98aa4f0c252a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:13.506040 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d99e44-a91b-46ad-91a1-98aa4f0c252a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:13.507482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d99e44-a91b-46ad-91a1-98aa4f0c252a None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:13.579151 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:13.599907 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 576/1159] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:45:13 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 23 13:45:13.606491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12ec981-f15a-49b6-85f9-35b17645f1cf None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:13.683136 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:13.701315 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12ec981-f15a-49b6-85f9-35b17645f1cf None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:13.736213 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64ffddb-ac5f-480f-907a-c55cb7c3196e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:13.736831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64ffddb-ac5f-480f-907a-c55cb7c3196e None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:13.737105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64ffddb-ac5f-480f-907a-c55cb7c3196e None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:13.737370 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64ffddb-ac5f-480f-907a-c55cb7c3196e None admin] PATH_INFO: `/v3/projects/30c69f3ee4ff4852b4e363b0d1b33fdb` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:13.738496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64ffddb-ac5f-480f-907a-c55cb7c3196e None admin] RBAC: Authorizing `identity:get_project(project_id=30c69f3ee4ff4852b4e363b0d1b33fdb)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:13.739908 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64ffddb-ac5f-480f-907a-c55cb7c3196e None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:13.741509 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 584/1160] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:45:13 2022] GET /identity/v3/projects/30c69f3ee4ff4852b4e363b0d1b33fdb => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:13.748004 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d99e44-a91b-46ad-91a1-98aa4f0c252a None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:13.842860 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d99e44-a91b-46ad-91a1-98aa4f0c252a None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:13.875433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0de2bc-4906-4dbd-b2d1-7fa910e7ce5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:13.876011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0de2bc-4906-4dbd-b2d1-7fa910e7ce5e None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:13.876283 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0de2bc-4906-4dbd-b2d1-7fa910e7ce5e None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:13.876547 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0de2bc-4906-4dbd-b2d1-7fa910e7ce5e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:13.877255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0de2bc-4906-4dbd-b2d1-7fa910e7ce5e None admin] RBAC: Adding query filter params (role.id=727f0fa598e54b9a929d2cf6277c875e, scope.project.id=30c69f3ee4ff4852b4e363b0d1b33fdb, user.id=f5d137418041465b9ad4de99e2059e2a) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:45:13.877554 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0de2bc-4906-4dbd-b2d1-7fa910e7ce5e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:13.878970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0de2bc-4906-4dbd-b2d1-7fa910e7ce5e None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:13.884444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 577/1161] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:45:13 2022] GET /identity/v3/role_assignments?user.id=f5d137418041465b9ad4de99e2059e2a&scope.project.id=30c69f3ee4ff4852b4e363b0d1b33fdb&role.id=727f0fa598e54b9a929d2cf6277c875e => generated 608 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:16.045538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a147b9a-1696-45a8-84dc-72860a6b831f None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:16.046175 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a147b9a-1696-45a8-84dc-72860a6b831f None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:16.046175 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a147b9a-1696-45a8-84dc-72860a6b831f None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:16.047449 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 585/1162] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:45:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:45:16.052780 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3c8c19-35f2-4e06-ab8e-ebed00ac3223 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:16.053206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3c8c19-35f2-4e06-ab8e-ebed00ac3223 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:16.053315 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3c8c19-35f2-4e06-ab8e-ebed00ac3223 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:16.071607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-6c3c8c19-35f2-4e06-ab8e-ebed00ac3223 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:45:16.072791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-6c3c8c19-35f2-4e06-ab8e-ebed00ac3223 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:45:16.106931 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:16.136871 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 578/1163] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:45:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:45:16.406717 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a147b9a-1696-45a8-84dc-72860a6b831f None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:16.505289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a147b9a-1696-45a8-84dc-72860a6b831f None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:16.538296 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd44d00-540d-427a-a289-a5204ffbf909 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:16.538918 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd44d00-540d-427a-a289-a5204ffbf909 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:16.539197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd44d00-540d-427a-a289-a5204ffbf909 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:16.539463 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd44d00-540d-427a-a289-a5204ffbf909 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:16.557027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:16.606667 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd44d00-540d-427a-a289-a5204ffbf909 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:16.608088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dd44d00-540d-427a-a289-a5204ffbf909 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:16.701121 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 586/1164] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:45:16 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 23 13:45:16.707192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3c8c19-35f2-4e06-ab8e-ebed00ac3223 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:16.803924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3c8c19-35f2-4e06-ab8e-ebed00ac3223 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:16.836255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a853bf-75ca-4a6a-8e25-6f6d1eb40923 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:16.836868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a853bf-75ca-4a6a-8e25-6f6d1eb40923 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:16.837147 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a853bf-75ca-4a6a-8e25-6f6d1eb40923 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:16.837417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a853bf-75ca-4a6a-8e25-6f6d1eb40923 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:16.844610 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a853bf-75ca-4a6a-8e25-6f6d1eb40923 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:16.846025 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a853bf-75ca-4a6a-8e25-6f6d1eb40923 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:16.857089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-59a853bf-75ca-4a6a-8e25-6f6d1eb40923 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 23 13:45:16.857089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:45:16.857089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 23 13:45:16.857089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 23 13:45:16.857089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:45:16.857089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:45:16.857089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:45:16.857089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:45:16.857089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 23 13:45:16.857089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:45:16.857089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:45:16.857089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:45:16.857089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:45:16.857089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:45:16.857089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:45:16.857089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:45:16.857089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:45:16.858506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:45:16.858506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:45:16.858506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:45:16.858506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:45:16.858506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:45:16.858506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:45:16.858506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:45:16.858506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:45:16.858506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:45:16.858506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:45:16.858506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:45:16.858506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:45:16.858506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:45:16.858506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:45:16.858506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:45:16.858506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:45:16.859652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:45:16.859652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:45:16.859652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:45:16.859652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:45:16.859652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:45:16.859652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:45:16.859652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:45:16.859652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:45:16.859652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:45:16.859652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:45:16.859652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 23 13:45:16.859652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:45:16.859652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 23 13:45:16.859652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:45:16.859652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 579/1165] 10.222.0.62 () {56 vars in 1117 bytes} [Fri Sep 23 13:45:16 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:45:16.864477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd44d00-540d-427a-a289-a5204ffbf909 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:16.911143 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:16.959379 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd44d00-540d-427a-a289-a5204ffbf909 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:16.991674 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab05aff-c943-454f-b5d3-1a30e7f3a985 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:16.992261 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab05aff-c943-454f-b5d3-1a30e7f3a985 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:16.992534 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab05aff-c943-454f-b5d3-1a30e7f3a985 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:16.992800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab05aff-c943-454f-b5d3-1a30e7f3a985 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:16.993426 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab05aff-c943-454f-b5d3-1a30e7f3a985 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:45:16.993717 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab05aff-c943-454f-b5d3-1a30e7f3a985 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:16.995074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab05aff-c943-454f-b5d3-1a30e7f3a985 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:17.002396 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 587/1166] 10.222.0.62 () {56 vars in 1115 bytes} [Fri Sep 23 13:45:16 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:19.111504 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d9bc65-9daa-48b9-9c8e-d06c49280d16 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:19.112067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d9bc65-9daa-48b9-9c8e-d06c49280d16 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:19.112067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d9bc65-9daa-48b9-9c8e-d06c49280d16 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:19.113390 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 580/1167] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:45:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:45:19.118511 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12129ac0-aaff-4359-87c9-a764955f1379 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:19.118900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12129ac0-aaff-4359-87c9-a764955f1379 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:19.119060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12129ac0-aaff-4359-87c9-a764955f1379 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:19.137286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-12129ac0-aaff-4359-87c9-a764955f1379 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:45:19.138507 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-12129ac0-aaff-4359-87c9-a764955f1379 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:45:19.202301 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 588/1168] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:45:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:45:19.469202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d9bc65-9daa-48b9-9c8e-d06c49280d16 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:19.500511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:19.569186 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d9bc65-9daa-48b9-9c8e-d06c49280d16 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:19.601884 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-906f03b8-8a13-446b-a003-4edd3468506a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:19.602505 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906f03b8-8a13-446b-a003-4edd3468506a None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:19.602783 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906f03b8-8a13-446b-a003-4edd3468506a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:19.603052 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-906f03b8-8a13-446b-a003-4edd3468506a None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:19.603719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906f03b8-8a13-446b-a003-4edd3468506a None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:19.605805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-906f03b8-8a13-446b-a003-4edd3468506a None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:19.698003 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 581/1169] 10.222.0.62 () {60 vars in 1138 bytes} [Fri Sep 23 13:45:19 2022] POST /identity/v3/domains => generated 267 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 13:45:21.871878 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45fcafc-ff8d-490e-b507-6e3677e9d0da None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:21.872412 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45fcafc-ff8d-490e-b507-6e3677e9d0da None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:21.872412 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45fcafc-ff8d-490e-b507-6e3677e9d0da None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:21.873760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 589/1170] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:45:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:45:21.878954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2252709a-85f3-48fa-b012-a4b0b051e016 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:21.879336 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2252709a-85f3-48fa-b012-a4b0b051e016 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:21.879498 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2252709a-85f3-48fa-b012-a4b0b051e016 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:21.897225 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-2252709a-85f3-48fa-b012-a4b0b051e016 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:45:21.898416 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2252709a-85f3-48fa-b012-a4b0b051e016 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:45:21.962065 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 582/1171] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:45:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:45:22.231415 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45fcafc-ff8d-490e-b507-6e3677e9d0da None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:22.275170 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:22.330871 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45fcafc-ff8d-490e-b507-6e3677e9d0da None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:22.363605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2ff9fb-c250-4642-b033-a1cb162bde26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:22.364225 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2ff9fb-c250-4642-b033-a1cb162bde26 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:22.364494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2ff9fb-c250-4642-b033-a1cb162bde26 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:22.364759 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2ff9fb-c250-4642-b033-a1cb162bde26 None admin] PATH_INFO: `/v3/domains/d7991655ac034d4683bf8b87d632a7c0` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:22.375514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2ff9fb-c250-4642-b033-a1cb162bde26 None admin] RBAC: Authorizing `identity:get_domain(domain_id=d7991655ac034d4683bf8b87d632a7c0)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:22.376939 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2ff9fb-c250-4642-b033-a1cb162bde26 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:22.378536 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 590/1172] 10.222.0.62 () {56 vars in 1183 bytes} [Fri Sep 23 13:45:22 2022] GET /identity/v3/domains/d7991655ac034d4683bf8b87d632a7c0 => generated 267 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:22.384408 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2252709a-85f3-48fa-b012-a4b0b051e016 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:22.409016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:22.483347 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2252709a-85f3-48fa-b012-a4b0b051e016 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:22.517325 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baeee4ab-e864-43f0-beb1-6179e3abcfcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:22.517917 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baeee4ab-e864-43f0-beb1-6179e3abcfcd None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:22.518190 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baeee4ab-e864-43f0-beb1-6179e3abcfcd None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:22.518482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baeee4ab-e864-43f0-beb1-6179e3abcfcd None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:22.519123 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baeee4ab-e864-43f0-beb1-6179e3abcfcd None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:22.520510 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baeee4ab-e864-43f0-beb1-6179e3abcfcd None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:22.653519 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 583/1173] 10.222.0.62 () {60 vars in 1141 bytes} [Fri Sep 23 13:45:22 2022] POST /identity/v3/projects => generated 361 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:45:24.821941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed50251b-1b95-4b00-a80e-c99fec2e65dc None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:24.822525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed50251b-1b95-4b00-a80e-c99fec2e65dc None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:24.822525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed50251b-1b95-4b00-a80e-c99fec2e65dc None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:24.823843 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 591/1174] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:45:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:45:24.829163 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c5212f-ddff-4bfd-9a68-a30968052e72 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:24.829576 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c5212f-ddff-4bfd-9a68-a30968052e72 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:24.829717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c5212f-ddff-4bfd-9a68-a30968052e72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:24.848330 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-f3c5212f-ddff-4bfd-9a68-a30968052e72 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:45:24.849511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f3c5212f-ddff-4bfd-9a68-a30968052e72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:45:24.923844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 584/1175] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:45:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:45:25.203386 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed50251b-1b95-4b00-a80e-c99fec2e65dc None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:25.304892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed50251b-1b95-4b00-a80e-c99fec2e65dc None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:25.306381 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:25.337501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec56437-43c7-4e0d-a635-e589ac553d0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:25.338110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec56437-43c7-4e0d-a635-e589ac553d0b None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:25.338921 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec56437-43c7-4e0d-a635-e589ac553d0b None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:25.339191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec56437-43c7-4e0d-a635-e589ac553d0b None admin] PATH_INFO: `/v3/domains/d7991655ac034d4683bf8b87d632a7c0` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:25.340520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec56437-43c7-4e0d-a635-e589ac553d0b None admin] RBAC: Authorizing `identity:get_domain(domain_id=d7991655ac034d4683bf8b87d632a7c0)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:25.342315 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec56437-43c7-4e0d-a635-e589ac553d0b None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:25.346355 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 592/1176] 10.222.0.62 () {56 vars in 1183 bytes} [Fri Sep 23 13:45:25 2022] GET /identity/v3/domains/d7991655ac034d4683bf8b87d632a7c0 => generated 267 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:25.349688 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c5212f-ddff-4bfd-9a68-a30968052e72 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:25.352718 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:25.446285 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c5212f-ddff-4bfd-9a68-a30968052e72 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:25.478906 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a7f25c-5c99-4d88-b16f-416474d64dbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:25.480038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a7f25c-5c99-4d88-b16f-416474d64dbc None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:25.480316 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a7f25c-5c99-4d88-b16f-416474d64dbc None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:25.480590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a7f25c-5c99-4d88-b16f-416474d64dbc None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:25.481218 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a7f25c-5c99-4d88-b16f-416474d64dbc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:25.482644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a7f25c-5c99-4d88-b16f-416474d64dbc None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:25.582070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 585/1177] 10.222.0.62 () {60 vars in 1132 bytes} [Fri Sep 23 13:45:25 2022] POST /identity/v3/users => generated 311 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:45:27.761672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7389cf21-2b59-434d-8abf-469e29fab090 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:27.762210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7389cf21-2b59-434d-8abf-469e29fab090 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:27.762210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7389cf21-2b59-434d-8abf-469e29fab090 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:27.763577 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 593/1178] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:45:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:45:27.768763 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4bc49a-4447-43d5-aa5b-6ca495675ded None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:27.769133 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4bc49a-4447-43d5-aa5b-6ca495675ded None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:27.769308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4bc49a-4447-43d5-aa5b-6ca495675ded None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:27.787658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-af4bc49a-4447-43d5-aa5b-6ca495675ded None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:45:27.788829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-af4bc49a-4447-43d5-aa5b-6ca495675ded None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:45:27.842441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:27.853027 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 586/1179] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:45:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:45:28.127122 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7389cf21-2b59-434d-8abf-469e29fab090 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:28.232996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7389cf21-2b59-434d-8abf-469e29fab090 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:28.267751 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e51de2b-082d-4ced-9b77-824024636447 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:28.268366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e51de2b-082d-4ced-9b77-824024636447 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:28.268642 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e51de2b-082d-4ced-9b77-824024636447 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:28.268908 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e51de2b-082d-4ced-9b77-824024636447 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:28.278948 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e51de2b-082d-4ced-9b77-824024636447 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:45:28.289374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-2e51de2b-082d-4ced-9b77-824024636447 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:45:28.289374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:45:28.289374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:45:28.289374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:45:28.289374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:45:28.289374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:45:28.289374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:45:28.289374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:45:28.289374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:45:28.289374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:45:28.289374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:45:28.289374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:45:28.289374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:45:28.289374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:45:28.289374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:45:28.289374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:45:28.290520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:45:28.290520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:45:28.290520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:45:28.290520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:45:28.290520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:45:28.290520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:45:28.290520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:45:28.290520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:45:28.290520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:45:28.290520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:45:28.290520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:45:28.290520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:45:28.290520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:45:28.290520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:45:28.290520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:45:28.290520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:45:28.291750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:45:28.291750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:45:28.291750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:45:28.291750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:45:28.291750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:45:28.291750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:45:28.291750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:45:28.291750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:45:28.291750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:45:28.291750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:45:28.291750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:45:28.291750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:45:28.291750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 594/1180] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:45:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:45:28.292815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4bc49a-4447-43d5-aa5b-6ca495675ded None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:28.386747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4bc49a-4447-43d5-aa5b-6ca495675ded None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:28.419394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165a9d64-561f-4240-ad9c-a0fff481f5e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:28.419984 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165a9d64-561f-4240-ad9c-a0fff481f5e4 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:28.420262 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165a9d64-561f-4240-ad9c-a0fff481f5e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:28.420529 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165a9d64-561f-4240-ad9c-a0fff481f5e4 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:28.421179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165a9d64-561f-4240-ad9c-a0fff481f5e4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:45:28.421479 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165a9d64-561f-4240-ad9c-a0fff481f5e4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:28.422865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165a9d64-561f-4240-ad9c-a0fff481f5e4 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:28.430546 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 587/1181] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:45:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:28.437020 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e51de2b-082d-4ced-9b77-824024636447 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:28.467225 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:28.534043 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e51de2b-082d-4ced-9b77-824024636447 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:28.566405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8aa4e3-a4c4-4629-81b7-718759fbc9a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:28.567012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8aa4e3-a4c4-4629-81b7-718759fbc9a6 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:28.567286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8aa4e3-a4c4-4629-81b7-718759fbc9a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:28.567558 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8aa4e3-a4c4-4629-81b7-718759fbc9a6 None admin] PATH_INFO: `/v3/users/53a9604ed4654cbeb1bca73ac64642c2` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:28.580461 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8aa4e3-a4c4-4629-81b7-718759fbc9a6 None admin] RBAC: Authorizing `identity:get_user(user_id=53a9604ed4654cbeb1bca73ac64642c2)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:28.581875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8aa4e3-a4c4-4629-81b7-718759fbc9a6 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:28.583759 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 595/1182] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:45:28 2022] GET /identity/v3/users/53a9604ed4654cbeb1bca73ac64642c2 => generated 311 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:28.589428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165a9d64-561f-4240-ad9c-a0fff481f5e4 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:28.641222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:28.687543 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165a9d64-561f-4240-ad9c-a0fff481f5e4 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:28.718915 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53764244-578f-4fce-8df1-5064aa1520cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:28.719502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53764244-578f-4fce-8df1-5064aa1520cd None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:28.719781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53764244-578f-4fce-8df1-5064aa1520cd None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:28.720049 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53764244-578f-4fce-8df1-5064aa1520cd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:28.784812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53764244-578f-4fce-8df1-5064aa1520cd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:28.800226 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53764244-578f-4fce-8df1-5064aa1520cd None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:28.835243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:28.880791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 588/1183] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:45:28 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 23 13:45:28.887301 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8aa4e3-a4c4-4629-81b7-718759fbc9a6 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:28.971512 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:28.983032 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8aa4e3-a4c4-4629-81b7-718759fbc9a6 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:29.016777 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f129cb-b283-4023-abb1-3efe7d932f4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:29.017389 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f129cb-b283-4023-abb1-3efe7d932f4a None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:29.017666 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f129cb-b283-4023-abb1-3efe7d932f4a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:29.017933 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f129cb-b283-4023-abb1-3efe7d932f4a None admin] PATH_INFO: `/v3/projects/67d3c8ef313644798644ac9189ef5fa2` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:29.019061 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f129cb-b283-4023-abb1-3efe7d932f4a None admin] RBAC: Authorizing `identity:get_project(project_id=67d3c8ef313644798644ac9189ef5fa2)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:29.020475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6f129cb-b283-4023-abb1-3efe7d932f4a None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:29.022076 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 596/1184] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:45:28 2022] GET /identity/v3/projects/67d3c8ef313644798644ac9189ef5fa2 => generated 361 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:29.028163 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53764244-578f-4fce-8df1-5064aa1520cd None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:29.123786 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53764244-578f-4fce-8df1-5064aa1520cd None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:29.157073 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34780679-7c77-4bee-b749-b8f5691256ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:29.157655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34780679-7c77-4bee-b749-b8f5691256ce None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:29.157927 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34780679-7c77-4bee-b749-b8f5691256ce None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:29.158190 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34780679-7c77-4bee-b749-b8f5691256ce None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:29.158925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34780679-7c77-4bee-b749-b8f5691256ce None admin] RBAC: Adding query filter params (role.id=727f0fa598e54b9a929d2cf6277c875e, scope.project.id=67d3c8ef313644798644ac9189ef5fa2, user.id=53a9604ed4654cbeb1bca73ac64642c2) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:45:29.159225 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34780679-7c77-4bee-b749-b8f5691256ce None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:29.160615 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34780679-7c77-4bee-b749-b8f5691256ce None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:29.165746 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 589/1185] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:45:29 2022] GET /identity/v3/role_assignments?user.id=53a9604ed4654cbeb1bca73ac64642c2&scope.project.id=67d3c8ef313644798644ac9189ef5fa2&role.id=727f0fa598e54b9a929d2cf6277c875e => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:31.269594 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccce516a-c632-4e44-86eb-4ae9ff2ceb60 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:31.270085 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccce516a-c632-4e44-86eb-4ae9ff2ceb60 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:31.270190 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccce516a-c632-4e44-86eb-4ae9ff2ceb60 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:31.271496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 597/1186] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:45:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:45:31.276674 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb07a22-83df-4f5b-b0a5-580d72d3a292 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:31.277082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb07a22-83df-4f5b-b0a5-580d72d3a292 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:31.277203 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb07a22-83df-4f5b-b0a5-580d72d3a292 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:31.294832 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-6cb07a22-83df-4f5b-b0a5-580d72d3a292 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:45:31.295992 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-6cb07a22-83df-4f5b-b0a5-580d72d3a292 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:45:31.306171 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:31.360030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 590/1187] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:45:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:45:31.637671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccce516a-c632-4e44-86eb-4ae9ff2ceb60 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:31.735684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccce516a-c632-4e44-86eb-4ae9ff2ceb60 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:31.768142 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898baf45-f921-4c56-92a3-c96be245ff71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:31.768747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898baf45-f921-4c56-92a3-c96be245ff71 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:31.769021 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898baf45-f921-4c56-92a3-c96be245ff71 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:31.769288 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898baf45-f921-4c56-92a3-c96be245ff71 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:31.779610 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-898baf45-f921-4c56-92a3-c96be245ff71 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:45:31.784279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-898baf45-f921-4c56-92a3-c96be245ff71 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:45:31.784279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:45:31.784279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:45:31.784279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:45:31.784279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:45:31.784279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:45:31.784279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:45:31.784279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:45:31.784279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:45:31.784279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:45:31.784279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:45:31.784279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:45:31.784279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:45:31.784279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:45:31.784279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:45:31.784279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:45:31.785363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:45:31.785363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:45:31.785363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:45:31.785363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:45:31.785363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:45:31.785363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:45:31.785363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:45:31.785363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:45:31.785363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:45:31.785363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:45:31.785363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:45:31.785363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:45:31.785363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:45:31.785363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:45:31.785363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:45:31.785363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:45:31.786231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:45:31.786231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:45:31.786231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:45:31.786231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:45:31.786231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:45:31.786231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:45:31.786231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:45:31.786231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:45:31.786231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:45:31.786231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:45:31.786231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:45:31.786231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:45:31.786231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 598/1188] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:45:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:45:31.791627 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb07a22-83df-4f5b-b0a5-580d72d3a292 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:31.887205 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb07a22-83df-4f5b-b0a5-580d72d3a292 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:31.920344 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01f06c9-8e79-467d-ad31-f340752308bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:31.920872 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01f06c9-8e79-467d-ad31-f340752308bb None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:31.921152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01f06c9-8e79-467d-ad31-f340752308bb None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:31.921419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01f06c9-8e79-467d-ad31-f340752308bb None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:31.922070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01f06c9-8e79-467d-ad31-f340752308bb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:45:31.922396 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01f06c9-8e79-467d-ad31-f340752308bb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:31.923736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01f06c9-8e79-467d-ad31-f340752308bb None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:31.931505 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 591/1189] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:45:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:45:31.937343 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898baf45-f921-4c56-92a3-c96be245ff71 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:31.955998 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:32.036598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898baf45-f921-4c56-92a3-c96be245ff71 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:32.068867 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446a2fda-b9a5-46e9-8545-4dabe9bb9bfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:32.069471 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446a2fda-b9a5-46e9-8545-4dabe9bb9bfe None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:32.069746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446a2fda-b9a5-46e9-8545-4dabe9bb9bfe None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:32.070012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446a2fda-b9a5-46e9-8545-4dabe9bb9bfe None admin] PATH_INFO: `/v3/users/53a9604ed4654cbeb1bca73ac64642c2` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:32.071127 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446a2fda-b9a5-46e9-8545-4dabe9bb9bfe None admin] RBAC: Authorizing `identity:get_user(user_id=53a9604ed4654cbeb1bca73ac64642c2)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:32.073147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446a2fda-b9a5-46e9-8545-4dabe9bb9bfe None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:32.074720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 599/1190] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:45:31 2022] GET /identity/v3/users/53a9604ed4654cbeb1bca73ac64642c2 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:32.080733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01f06c9-8e79-467d-ad31-f340752308bb None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:32.091894 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:32.177742 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01f06c9-8e79-467d-ad31-f340752308bb None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:32.213425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2044e7-9f8e-4cfe-bb94-da7a371f50c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:32.214052 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2044e7-9f8e-4cfe-bb94-da7a371f50c5 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:32.214765 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2044e7-9f8e-4cfe-bb94-da7a371f50c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:32.214765 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2044e7-9f8e-4cfe-bb94-da7a371f50c5 None admin] PATH_INFO: `/v3/projects/67d3c8ef313644798644ac9189ef5fa2` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:32.215711 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2044e7-9f8e-4cfe-bb94-da7a371f50c5 None admin] RBAC: Authorizing `identity:get_project(project_id=67d3c8ef313644798644ac9189ef5fa2)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:32.217128 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2044e7-9f8e-4cfe-bb94-da7a371f50c5 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:32.218792 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 592/1191] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:45:32 2022] GET /identity/v3/projects/67d3c8ef313644798644ac9189ef5fa2 => generated 361 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:32.224792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446a2fda-b9a5-46e9-8545-4dabe9bb9bfe None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:32.310005 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:32.326404 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446a2fda-b9a5-46e9-8545-4dabe9bb9bfe None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:32.360930 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133b5d16-16c2-4fa6-a67c-603a50e8ba04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:32.361576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133b5d16-16c2-4fa6-a67c-603a50e8ba04 None admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:32.361856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133b5d16-16c2-4fa6-a67c-603a50e8ba04 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:32.362129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133b5d16-16c2-4fa6-a67c-603a50e8ba04 None admin] PATH_INFO: `/v3/projects/67d3c8ef313644798644ac9189ef5fa2/users/53a9604ed4654cbeb1bca73ac64642c2/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:32.364041 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133b5d16-16c2-4fa6-a67c-603a50e8ba04 None admin] RBAC: Authorizing `identity:create_grant(project_id=67d3c8ef313644798644ac9189ef5fa2, user_id=53a9604ed4654cbeb1bca73ac64642c2, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:32.365464 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133b5d16-16c2-4fa6-a67c-603a50e8ba04 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:32.440066 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 600/1192] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:45:32 2022] PUT /identity/v3/projects/67d3c8ef313644798644ac9189ef5fa2/users/53a9604ed4654cbeb1bca73ac64642c2/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:45:34.627586 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdd4d0f-ceeb-458e-8c3c-5de3b0e1ac29 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:34.628084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdd4d0f-ceeb-458e-8c3c-5de3b0e1ac29 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:34.628084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebdd4d0f-ceeb-458e-8c3c-5de3b0e1ac29 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:34.629378 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 593/1193] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:45:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:45:34.635346 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40d42dc-99fd-4fcd-a4f5-284b78bf5f6d None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:34.635845 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40d42dc-99fd-4fcd-a4f5-284b78bf5f6d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:34.635938 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40d42dc-99fd-4fcd-a4f5-284b78bf5f6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:34.654065 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-e40d42dc-99fd-4fcd-a4f5-284b78bf5f6d None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:45:34.655260 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-e40d42dc-99fd-4fcd-a4f5-284b78bf5f6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:45:34.718616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 601/1194] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:45:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:45:34.987381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdd4d0f-ceeb-458e-8c3c-5de3b0e1ac29 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:35.063605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:35.086794 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebdd4d0f-ceeb-458e-8c3c-5de3b0e1ac29 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:35.119795 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a9c349-f4e7-4694-887c-51687a513135 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:35.120394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a9c349-f4e7-4694-887c-51687a513135 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:35.120671 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a9c349-f4e7-4694-887c-51687a513135 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:35.120939 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a9c349-f4e7-4694-887c-51687a513135 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:35.131121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a9c349-f4e7-4694-887c-51687a513135 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:45:35.136379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-b1a9c349-f4e7-4694-887c-51687a513135 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:45:35.136379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:45:35.136379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:45:35.136379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:45:35.136379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:45:35.136379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:45:35.136379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:45:35.136379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:45:35.136379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:45:35.136379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:45:35.136379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:45:35.136379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:45:35.136379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:45:35.136379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:45:35.136379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:45:35.136379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:45:35.137499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:45:35.137499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:45:35.137499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:45:35.137499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:45:35.137499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:45:35.137499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:45:35.137499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:45:35.137499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:45:35.137499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:45:35.137499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:45:35.137499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:45:35.137499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:45:35.137499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:45:35.137499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:45:35.137499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:45:35.137499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:45:35.138550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:45:35.138550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:45:35.138550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:45:35.138550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:45:35.138550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:45:35.138550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:45:35.138550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:45:35.138550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:45:35.138550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:45:35.138550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:45:35.138550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:45:35.138550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:45:35.138550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 594/1195] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:45:34 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:45:35.143633 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40d42dc-99fd-4fcd-a4f5-284b78bf5f6d None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:35.215961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:35.239661 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40d42dc-99fd-4fcd-a4f5-284b78bf5f6d None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:35.273490 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35b2f46-a1ed-4e2b-8834-88b4754f3923 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:35.274089 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35b2f46-a1ed-4e2b-8834-88b4754f3923 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:35.274393 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35b2f46-a1ed-4e2b-8834-88b4754f3923 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:35.274665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35b2f46-a1ed-4e2b-8834-88b4754f3923 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:35.275534 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35b2f46-a1ed-4e2b-8834-88b4754f3923 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:45:35.275832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35b2f46-a1ed-4e2b-8834-88b4754f3923 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:35.277185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35b2f46-a1ed-4e2b-8834-88b4754f3923 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:35.284937 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 602/1196] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:45:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:35.290960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a9c349-f4e7-4694-887c-51687a513135 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:35.388136 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a9c349-f4e7-4694-887c-51687a513135 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:35.420660 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba5ea22-2fd5-4b60-ac99-1dbb185f16a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:35.421274 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba5ea22-2fd5-4b60-ac99-1dbb185f16a6 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:35.421553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba5ea22-2fd5-4b60-ac99-1dbb185f16a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:35.421823 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba5ea22-2fd5-4b60-ac99-1dbb185f16a6 None admin] PATH_INFO: `/v3/users/53a9604ed4654cbeb1bca73ac64642c2` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:35.422934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba5ea22-2fd5-4b60-ac99-1dbb185f16a6 None admin] RBAC: Authorizing `identity:get_user(user_id=53a9604ed4654cbeb1bca73ac64642c2)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:35.424337 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba5ea22-2fd5-4b60-ac99-1dbb185f16a6 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:35.425888 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 595/1197] 10.222.0.62 () {56 vars in 1177 bytes} [Fri Sep 23 13:45:35 2022] GET /identity/v3/users/53a9604ed4654cbeb1bca73ac64642c2 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:45:35.432153 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35b2f46-a1ed-4e2b-8834-88b4754f3923 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:35.528893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35b2f46-a1ed-4e2b-8834-88b4754f3923 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:35.561384 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db0677d-5c7e-460f-aeb3-c428ea8c6b6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:35.561964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db0677d-5c7e-460f-aeb3-c428ea8c6b6c None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:35.562238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db0677d-5c7e-460f-aeb3-c428ea8c6b6c None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:35.562535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db0677d-5c7e-460f-aeb3-c428ea8c6b6c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:35.565715 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:35.627221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db0677d-5c7e-460f-aeb3-c428ea8c6b6c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:35.628635 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db0677d-5c7e-460f-aeb3-c428ea8c6b6c None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:35.722345 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 603/1198] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:45:35 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 23 13:45:35.729513 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba5ea22-2fd5-4b60-ac99-1dbb185f16a6 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:35.733494 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:35.826806 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba5ea22-2fd5-4b60-ac99-1dbb185f16a6 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:35.859537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdea4f2-11c5-4943-8bfe-43f23273abbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:35.860150 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdea4f2-11c5-4943-8bfe-43f23273abbe None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:35.860423 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdea4f2-11c5-4943-8bfe-43f23273abbe None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:35.860692 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdea4f2-11c5-4943-8bfe-43f23273abbe None admin] PATH_INFO: `/v3/projects/67d3c8ef313644798644ac9189ef5fa2` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:35.861797 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abdea4f2-11c5-4943-8bfe-43f23273abbe None admin] RBAC: Authorizing `identity:get_project(project_id=67d3c8ef313644798644ac9189ef5fa2)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:35.863228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abdea4f2-11c5-4943-8bfe-43f23273abbe None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:35.864834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 596/1199] 10.222.0.62 () {56 vars in 1186 bytes} [Fri Sep 23 13:45:35 2022] GET /identity/v3/projects/67d3c8ef313644798644ac9189ef5fa2 => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:35.871020 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db0677d-5c7e-460f-aeb3-c428ea8c6b6c None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:45:35.904262 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:35.966837 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db0677d-5c7e-460f-aeb3-c428ea8c6b6c None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:45:36.000167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c83f1b-21bb-48f1-9b97-1bf53ffa34ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:45:36.000746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c83f1b-21bb-48f1-9b97-1bf53ffa34ff None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:45:36.001018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c83f1b-21bb-48f1-9b97-1bf53ffa34ff None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:45:36.001280 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c83f1b-21bb-48f1-9b97-1bf53ffa34ff None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:45:36.001983 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c83f1b-21bb-48f1-9b97-1bf53ffa34ff None admin] RBAC: Adding query filter params (role.id=727f0fa598e54b9a929d2cf6277c875e, scope.project.id=67d3c8ef313644798644ac9189ef5fa2, user.id=53a9604ed4654cbeb1bca73ac64642c2) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:45:36.002280 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c83f1b-21bb-48f1-9b97-1bf53ffa34ff None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:45:36.003789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c83f1b-21bb-48f1-9b97-1bf53ffa34ff None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:45:36.009266 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 604/1200] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:45:35 2022] GET /identity/v3/role_assignments?user.id=53a9604ed4654cbeb1bca73ac64642c2&scope.project.id=67d3c8ef313644798644ac9189ef5fa2&role.id=727f0fa598e54b9a929d2cf6277c875e => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:45:45.855206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=64 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:45:46.006815 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=50 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:46:48.103268 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb27dff-2c3a-47e8-b7a7-b39193e68cbb None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:48.103926 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb27dff-2c3a-47e8-b7a7-b39193e68cbb None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:48.103926 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb27dff-2c3a-47e8-b7a7-b39193e68cbb None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:48.105196 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 597/1201] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:46:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:46:48.110573 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b802c500-6264-4640-8e39-16bb7179bb02 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:48.117258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b802c500-6264-4640-8e39-16bb7179bb02 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:48.117258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b802c500-6264-4640-8e39-16bb7179bb02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:48.129554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-b802c500-6264-4640-8e39-16bb7179bb02 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:46:48.130773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-b802c500-6264-4640-8e39-16bb7179bb02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:46:48.198654 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 605/1202] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:46:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:46:48.476095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb27dff-2c3a-47e8-b7a7-b39193e68cbb None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:48.580355 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb27dff-2c3a-47e8-b7a7-b39193e68cbb None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:48.614079 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbe4818-4bbd-4bfc-a1ca-3df5f19b28fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:48.616372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbe4818-4bbd-4bfc-a1ca-3df5f19b28fd None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:48.616372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbe4818-4bbd-4bfc-a1ca-3df5f19b28fd None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:48.616372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbe4818-4bbd-4bfc-a1ca-3df5f19b28fd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:48.625277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bbe4818-4bbd-4bfc-a1ca-3df5f19b28fd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:48.626721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bbe4818-4bbd-4bfc-a1ca-3df5f19b28fd None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:48.641029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-9bbe4818-4bbd-4bfc-a1ca-3df5f19b28fd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:46:48.641029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:46:48.641029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:46:48.641029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:46:48.641029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:46:48.641029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:46:48.641029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:46:48.641029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:46:48.641029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:46:48.641029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:46:48.641029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:46:48.641029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:46:48.641029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:46:48.641029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:46:48.641029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:46:48.641029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:46:48.642250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:46:48.642250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:46:48.642250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:46:48.642250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:46:48.642250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:46:48.642250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:46:48.642250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:46:48.642250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:46:48.642250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:46:48.642250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:46:48.642250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:46:48.642250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:46:48.642250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:46:48.642250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:46:48.642250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:46:48.642250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:46:48.642250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:46:48.643641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:46:48.643641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:46:48.643641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:46:48.643641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 598/1203] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:46:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:46:48.648313 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b802c500-6264-4640-8e39-16bb7179bb02 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:48.745638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b802c500-6264-4640-8e39-16bb7179bb02 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:48.773852 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:46:48.779613 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd70321-aedf-4a9d-8292-71151d63c2e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:48.780216 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd70321-aedf-4a9d-8292-71151d63c2e3 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:48.780505 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd70321-aedf-4a9d-8292-71151d63c2e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:48.780772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd70321-aedf-4a9d-8292-71151d63c2e3 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:48.781408 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd70321-aedf-4a9d-8292-71151d63c2e3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:46:48.781708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd70321-aedf-4a9d-8292-71151d63c2e3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:48.783097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd70321-aedf-4a9d-8292-71151d63c2e3 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:48.794487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 606/1204] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:46:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:46:48.800206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbe4818-4bbd-4bfc-a1ca-3df5f19b28fd None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:48.853235 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:46:48.899133 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbe4818-4bbd-4bfc-a1ca-3df5f19b28fd None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:48.932134 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91e57838-d72d-406b-9955-d2742843c62c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:48.932731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e57838-d72d-406b-9955-d2742843c62c None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:48.933009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e57838-d72d-406b-9955-d2742843c62c None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:48.933279 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e57838-d72d-406b-9955-d2742843c62c None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:48.933914 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e57838-d72d-406b-9955-d2742843c62c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:48.935350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91e57838-d72d-406b-9955-d2742843c62c None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:49.008650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 599/1205] 10.222.0.62 () {60 vars in 1132 bytes} [Fri Sep 23 13:46:48 2022] POST /identity/v3/users => generated 251 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:46:51.185789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cbfdd6-1d1c-453b-819b-b10511093c74 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:51.186365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cbfdd6-1d1c-453b-819b-b10511093c74 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:51.186365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cbfdd6-1d1c-453b-819b-b10511093c74 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:51.187738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 607/1206] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:46:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:46:51.193078 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c55ae9-0b10-4c50-ba4f-327821b82952 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:51.193482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c55ae9-0b10-4c50-ba4f-327821b82952 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:51.193613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c55ae9-0b10-4c50-ba4f-327821b82952 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:51.212608 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-76c55ae9-0b10-4c50-ba4f-327821b82952 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:46:51.214764 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-76c55ae9-0b10-4c50-ba4f-327821b82952 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:46:51.280391 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 600/1207] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:46:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:46:51.551513 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cbfdd6-1d1c-453b-819b-b10511093c74 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:51.652897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cbfdd6-1d1c-453b-819b-b10511093c74 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:51.685660 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9feeafd-e7c7-4e75-a18b-3510ccbcecd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:51.686305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9feeafd-e7c7-4e75-a18b-3510ccbcecd8 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:51.686610 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9feeafd-e7c7-4e75-a18b-3510ccbcecd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:51.686874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9feeafd-e7c7-4e75-a18b-3510ccbcecd8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:51.697440 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9feeafd-e7c7-4e75-a18b-3510ccbcecd8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:46:51.702285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-c9feeafd-e7c7-4e75-a18b-3510ccbcecd8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:46:51.702285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:46:51.702285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:46:51.702285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:46:51.702285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:46:51.702285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:46:51.702285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:46:51.702285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:46:51.702285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:46:51.702285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:46:51.702285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:46:51.702285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:46:51.702285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:46:51.702285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:46:51.702285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:46:51.702285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:46:51.703394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:46:51.703394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:46:51.703394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:46:51.703394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:46:51.703394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:46:51.703394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:46:51.703394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:46:51.703394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:46:51.703394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:46:51.703394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:46:51.703394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:46:51.703394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:46:51.703394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:46:51.703394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:46:51.703394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:46:51.703394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:46:51.704235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:46:51.704235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:46:51.704235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:46:51.704235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:46:51.704235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:46:51.704235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:46:51.704235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:46:51.704235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:46:51.704235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:46:51.704235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:46:51.704235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:46:51.704235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:46:51.704235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 608/1208] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:46:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:46:51.709734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c55ae9-0b10-4c50-ba4f-327821b82952 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:51.748296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:46:51.806158 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c55ae9-0b10-4c50-ba4f-327821b82952 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:51.838790 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c70c95d-6b0f-480a-9411-6eccdc90bbe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:51.839388 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c70c95d-6b0f-480a-9411-6eccdc90bbe5 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:51.839668 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c70c95d-6b0f-480a-9411-6eccdc90bbe5 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:51.839937 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c70c95d-6b0f-480a-9411-6eccdc90bbe5 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:51.840589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c70c95d-6b0f-480a-9411-6eccdc90bbe5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:46:51.840888 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c70c95d-6b0f-480a-9411-6eccdc90bbe5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:51.842239 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c70c95d-6b0f-480a-9411-6eccdc90bbe5 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:51.850636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 601/1209] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:46:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:46:51.856763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9feeafd-e7c7-4e75-a18b-3510ccbcecd8 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:51.907489 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:46:51.956442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9feeafd-e7c7-4e75-a18b-3510ccbcecd8 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:51.991058 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0bed1a-1afb-4e61-97a8-3463634f9cb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:51.991678 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0bed1a-1afb-4e61-97a8-3463634f9cb1 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:51.991953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0bed1a-1afb-4e61-97a8-3463634f9cb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:51.992218 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0bed1a-1afb-4e61-97a8-3463634f9cb1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:52.001813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0bed1a-1afb-4e61-97a8-3463634f9cb1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:52.007339 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0bed1a-1afb-4e61-97a8-3463634f9cb1 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:52.017691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-6d0bed1a-1afb-4e61-97a8-3463634f9cb1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:46:52.017691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:46:52.017691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:46:52.017691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:46:52.017691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:46:52.017691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:46:52.017691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:46:52.017691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:46:52.017691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:46:52.017691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:46:52.017691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:46:52.017691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:46:52.017691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:46:52.017691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:46:52.017691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:46:52.017691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:46:52.018827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:46:52.018827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:46:52.018827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:46:52.018827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:46:52.018827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:46:52.018827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:46:52.018827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:46:52.018827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:46:52.018827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:46:52.018827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:46:52.018827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:46:52.018827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:46:52.018827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:46:52.018827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:46:52.018827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:46:52.018827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:46:52.018827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:46:52.019923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:46:52.019923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:46:52.019923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:46:52.022479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 609/1210] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:46:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:46:52.024699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c70c95d-6b0f-480a-9411-6eccdc90bbe5 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:52.122549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:46:52.125286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c70c95d-6b0f-480a-9411-6eccdc90bbe5 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:52.158335 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322a603a-bfe6-402a-952b-326412629d3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:52.158954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322a603a-bfe6-402a-952b-326412629d3b None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:52.159235 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322a603a-bfe6-402a-952b-326412629d3b None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:52.159508 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322a603a-bfe6-402a-952b-326412629d3b None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:52.160768 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322a603a-bfe6-402a-952b-326412629d3b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:46:52.161051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322a603a-bfe6-402a-952b-326412629d3b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:52.162605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322a603a-bfe6-402a-952b-326412629d3b None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:52.174086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 602/1211] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:46:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:46:52.180168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0bed1a-1afb-4e61-97a8-3463634f9cb1 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:52.282600 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0bed1a-1afb-4e61-97a8-3463634f9cb1 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:52.284146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:46:52.315776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cdd45e-e181-4228-9d5e-963c7835dd59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:52.316844 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cdd45e-e181-4228-9d5e-963c7835dd59 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:52.316844 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cdd45e-e181-4228-9d5e-963c7835dd59 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:52.317132 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cdd45e-e181-4228-9d5e-963c7835dd59 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:52.324056 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cdd45e-e181-4228-9d5e-963c7835dd59 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:52.325482 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cdd45e-e181-4228-9d5e-963c7835dd59 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:52.337405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-a7cdd45e-e181-4228-9d5e-963c7835dd59 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 23 13:46:52.337405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:46:52.337405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:46:52.337405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:46:52.337405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:46:52.337405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:46:52.337405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:46:52.337405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:46:52.337405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:46:52.337405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:46:52.337405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:46:52.337405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:46:52.337405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:46:52.337405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:46:52.337405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:46:52.337405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:46:52.338617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:46:52.338617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:46:52.338617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:46:52.338617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:46:52.338617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:46:52.338617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:46:52.338617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:46:52.338617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:46:52.338617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:46:52.338617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:46:52.338617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:46:52.338617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:46:52.338617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:46:52.338617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:46:52.338617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:46:52.338617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:46:52.338617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:46:52.338617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:46:52.339866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:46:52.339866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:46:52.339866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:46:52.339866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:46:52.339866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:46:52.339866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:46:52.339866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:46:52.339866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:46:52.339866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 23 13:46:52.339866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:46:52.339866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 610/1212] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:46:52 2022] GET /identity/v3/users/placement => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:46:52.344647 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322a603a-bfe6-402a-952b-326412629d3b None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:52.455722 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322a603a-bfe6-402a-952b-326412629d3b None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:52.496931 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76174366-cfb0-4269-932e-0cfae2b3d012 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:52.497541 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76174366-cfb0-4269-932e-0cfae2b3d012 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:52.497818 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76174366-cfb0-4269-932e-0cfae2b3d012 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:52.498086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76174366-cfb0-4269-932e-0cfae2b3d012 None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:52.498842 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76174366-cfb0-4269-932e-0cfae2b3d012 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:46:52.499143 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76174366-cfb0-4269-932e-0cfae2b3d012 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:52.500564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76174366-cfb0-4269-932e-0cfae2b3d012 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:52.518580 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 603/1213] 10.222.0.62 () {56 vars in 1143 bytes} [Fri Sep 23 13:46:52 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:46:52.525382 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cdd45e-e181-4228-9d5e-963c7835dd59 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:52.623282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cdd45e-e181-4228-9d5e-963c7835dd59 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:52.656177 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3582b6db-487a-4249-8a46-1ecfcb69ce6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:52.656773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3582b6db-487a-4249-8a46-1ecfcb69ce6b None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:52.657051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3582b6db-487a-4249-8a46-1ecfcb69ce6b None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:52.657320 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3582b6db-487a-4249-8a46-1ecfcb69ce6b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:52.686019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:46:52.723795 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3582b6db-487a-4249-8a46-1ecfcb69ce6b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:52.725458 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3582b6db-487a-4249-8a46-1ecfcb69ce6b None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:52.823618 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 611/1214] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:46:52 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 23 13:46:52.829560 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76174366-cfb0-4269-932e-0cfae2b3d012 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:52.848537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:46:52.925756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76174366-cfb0-4269-932e-0cfae2b3d012 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:52.959759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6e85de-6b56-449a-8bc5-75e35ce32b36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:52.960385 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6e85de-6b56-449a-8bc5-75e35ce32b36 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:52.960661 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6e85de-6b56-449a-8bc5-75e35ce32b36 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:52.960929 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6e85de-6b56-449a-8bc5-75e35ce32b36 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:52.971041 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a6e85de-6b56-449a-8bc5-75e35ce32b36 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:52.972472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a6e85de-6b56-449a-8bc5-75e35ce32b36 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:52.986970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-1a6e85de-6b56-449a-8bc5-75e35ce32b36 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:46:52.986970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:46:52.986970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:46:52.986970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:46:52.986970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:46:52.986970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:46:52.986970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:46:52.986970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:46:52.986970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:46:52.986970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:46:52.986970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:46:52.986970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:46:52.986970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:46:52.986970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:46:52.986970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:46:52.986970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:46:52.989044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:46:52.989044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:46:52.989044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:46:52.989044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:46:52.989044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:46:52.989044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:46:52.989044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:46:52.989044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:46:52.989044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:46:52.989044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:46:52.989044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:46:52.989044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:46:52.989044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:46:52.989044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:46:52.989044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:46:52.989044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:46:52.989044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:46:52.990226 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:46:52.990226 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:46:52.990226 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:46:52.990226 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 604/1215] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:46:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:46:52.994428 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3582b6db-487a-4249-8a46-1ecfcb69ce6b None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:53.054492 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:46:53.092902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3582b6db-487a-4249-8a46-1ecfcb69ce6b None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:53.126764 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fb8100-903c-4d0a-a153-caf7ebb6434b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:53.127367 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fb8100-903c-4d0a-a153-caf7ebb6434b None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:53.127643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fb8100-903c-4d0a-a153-caf7ebb6434b None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:53.127912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2fb8100-903c-4d0a-a153-caf7ebb6434b None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:53.129150 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2fb8100-903c-4d0a-a153-caf7ebb6434b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:46:53.129447 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2fb8100-903c-4d0a-a153-caf7ebb6434b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:53.130831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2fb8100-903c-4d0a-a153-caf7ebb6434b None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:53.142233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 612/1216] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:46:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:46:53.148145 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6e85de-6b56-449a-8bc5-75e35ce32b36 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:53.218055 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:46:53.249049 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6e85de-6b56-449a-8bc5-75e35ce32b36 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:53.283559 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bedd9f-297d-4a9b-af73-d98b989b9aa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:53.284181 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bedd9f-297d-4a9b-af73-d98b989b9aa1 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:53.284462 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bedd9f-297d-4a9b-af73-d98b989b9aa1 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:53.285668 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bedd9f-297d-4a9b-af73-d98b989b9aa1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:53.292208 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47bedd9f-297d-4a9b-af73-d98b989b9aa1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:53.293707 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47bedd9f-297d-4a9b-af73-d98b989b9aa1 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:53.306637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-47bedd9f-297d-4a9b-af73-d98b989b9aa1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:46:53.306637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:46:53.306637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:46:53.306637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:46:53.306637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:46:53.306637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:46:53.306637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:46:53.306637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:46:53.306637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:46:53.306637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:46:53.306637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:46:53.306637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:46:53.306637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:46:53.306637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:46:53.306637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:46:53.307745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:46:53.307745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:46:53.307745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:46:53.307745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:46:53.307745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:46:53.307745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:46:53.307745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:46:53.307745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:46:53.307745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:46:53.307745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:46:53.307745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:46:53.307745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:46:53.307745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:46:53.307745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:46:53.307745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:46:53.307745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:46:53.307745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:46:53.308858 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:46:53.308858 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:46:53.308858 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:46:53.308858 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:46:53.308858 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:46:53.308858 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:46:53.308858 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:46:53.308858 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:46:53.308858 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 605/1217] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:46:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:46:53.313961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fb8100-903c-4d0a-a153-caf7ebb6434b None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:53.412868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2fb8100-903c-4d0a-a153-caf7ebb6434b None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:53.435231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:46:53.446804 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d528c623-d429-4921-b520-e87fac64bb41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:53.447402 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d528c623-d429-4921-b520-e87fac64bb41 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:53.447678 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d528c623-d429-4921-b520-e87fac64bb41 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:53.447943 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d528c623-d429-4921-b520-e87fac64bb41 None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:53.448627 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d528c623-d429-4921-b520-e87fac64bb41 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:46:53.448929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d528c623-d429-4921-b520-e87fac64bb41 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:53.450376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d528c623-d429-4921-b520-e87fac64bb41 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:53.461694 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 613/1218] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:46:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:46:53.467976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bedd9f-297d-4a9b-af73-d98b989b9aa1 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:53.572952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bedd9f-297d-4a9b-af73-d98b989b9aa1 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:53.600681 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:46:53.606320 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-932542d8-7ad3-4784-b01d-af7a2fef6217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:53.606959 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932542d8-7ad3-4784-b01d-af7a2fef6217 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:53.607233 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932542d8-7ad3-4784-b01d-af7a2fef6217 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:53.607497 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932542d8-7ad3-4784-b01d-af7a2fef6217 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:53.608199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932542d8-7ad3-4784-b01d-af7a2fef6217 None admin] RBAC: Adding query filter params (role.id=809612fc6d62442cbcc179b5c4123465, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=ce249d4a99c34891b6ba941f57a14dd4) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:46:53.608498 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932542d8-7ad3-4784-b01d-af7a2fef6217 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:53.609896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-932542d8-7ad3-4784-b01d-af7a2fef6217 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:53.615818 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 606/1219] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:46:53 2022] GET /identity/v3/role_assignments?user.id=ce249d4a99c34891b6ba941f57a14dd4&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=809612fc6d62442cbcc179b5c4123465 => generated 259 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:46:55.740873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c11996-a51a-405c-8528-4ffc623cb0b3 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:55.741397 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c11996-a51a-405c-8528-4ffc623cb0b3 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:55.741397 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c11996-a51a-405c-8528-4ffc623cb0b3 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:55.742796 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 614/1220] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:46:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:46:55.747969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06300ffe-a711-411b-8249-d95dbebe664e None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:55.748341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06300ffe-a711-411b-8249-d95dbebe664e None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:55.748504 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06300ffe-a711-411b-8249-d95dbebe664e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:55.767029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-06300ffe-a711-411b-8249-d95dbebe664e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:46:55.768223 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-06300ffe-a711-411b-8249-d95dbebe664e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:46:55.833570 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 607/1221] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:46:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:46:56.105640 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c11996-a51a-405c-8528-4ffc623cb0b3 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:56.211359 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c11996-a51a-405c-8528-4ffc623cb0b3 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:56.247571 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07282424-f007-4779-a084-46ba3b454f32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:56.248196 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07282424-f007-4779-a084-46ba3b454f32 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:56.248497 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07282424-f007-4779-a084-46ba3b454f32 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:56.248768 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07282424-f007-4779-a084-46ba3b454f32 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:56.260127 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07282424-f007-4779-a084-46ba3b454f32 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:46:56.265697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-07282424-f007-4779-a084-46ba3b454f32 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:46:56.265697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:46:56.265697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:46:56.265697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:46:56.265697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:46:56.265697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:46:56.265697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:46:56.265697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:46:56.265697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:46:56.265697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:46:56.265697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:46:56.265697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:46:56.265697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:46:56.265697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:46:56.265697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:46:56.265697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:46:56.266830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:46:56.266830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:46:56.266830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:46:56.266830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:46:56.266830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:46:56.266830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:46:56.266830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:46:56.266830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:46:56.266830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:46:56.266830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:46:56.266830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:46:56.266830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:46:56.266830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:46:56.266830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:46:56.266830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:46:56.266830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:46:56.267868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:46:56.267868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:46:56.267868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:46:56.267868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:46:56.267868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:46:56.267868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:46:56.267868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:46:56.267868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:46:56.267868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:46:56.267868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:46:56.267868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:46:56.267868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:46:56.267868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 615/1222] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:46:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:46:56.272503 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06300ffe-a711-411b-8249-d95dbebe664e None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:56.379316 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06300ffe-a711-411b-8249-d95dbebe664e None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:56.401356 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:46:56.412182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22048621-f7bb-4245-a305-55611983f948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:56.412784 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22048621-f7bb-4245-a305-55611983f948 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:56.413062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22048621-f7bb-4245-a305-55611983f948 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:56.413331 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22048621-f7bb-4245-a305-55611983f948 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:56.413988 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22048621-f7bb-4245-a305-55611983f948 None admin] RBAC: Adding query filter params (name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:46:56.414286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22048621-f7bb-4245-a305-55611983f948 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:56.415668 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22048621-f7bb-4245-a305-55611983f948 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:56.423377 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 608/1223] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:46:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:46:56.429776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07282424-f007-4779-a084-46ba3b454f32 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:56.474529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:46:56.530900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07282424-f007-4779-a084-46ba3b454f32 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:56.566130 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef0650e-42e0-4429-81f5-206d36e6bf5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:56.566712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef0650e-42e0-4429-81f5-206d36e6bf5a None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:56.566992 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef0650e-42e0-4429-81f5-206d36e6bf5a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:56.567262 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef0650e-42e0-4429-81f5-206d36e6bf5a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:56.577051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef0650e-42e0-4429-81f5-206d36e6bf5a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:56.578513 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef0650e-42e0-4429-81f5-206d36e6bf5a None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:56.592900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-4ef0650e-42e0-4429-81f5-206d36e6bf5a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:46:56.592900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:46:56.592900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:46:56.592900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:46:56.592900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:46:56.592900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:46:56.592900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:46:56.592900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:46:56.592900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:46:56.592900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:46:56.592900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:46:56.592900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:46:56.592900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:46:56.592900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:46:56.592900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:46:56.592900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:46:56.594042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:46:56.594042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:46:56.594042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:46:56.594042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:46:56.594042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:46:56.594042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:46:56.594042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:46:56.594042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:46:56.594042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:46:56.594042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:46:56.594042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:46:56.594042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:46:56.594042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:46:56.594042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:46:56.594042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:46:56.594042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:46:56.594042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:46:56.595168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:46:56.595168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:46:56.595168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:46:56.595168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 616/1224] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:46:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:46:56.600821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22048621-f7bb-4245-a305-55611983f948 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:56.697662 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22048621-f7bb-4245-a305-55611983f948 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:56.730641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3e1efb-76fb-4715-801b-3688278f576d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:56.731231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3e1efb-76fb-4715-801b-3688278f576d None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:56.731510 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3e1efb-76fb-4715-801b-3688278f576d None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:56.731779 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3e1efb-76fb-4715-801b-3688278f576d None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:56.732410 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3e1efb-76fb-4715-801b-3688278f576d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:46:56.732710 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3e1efb-76fb-4715-801b-3688278f576d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:56.734057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3e1efb-76fb-4715-801b-3688278f576d None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:56.745609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 609/1225] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:46:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:46:56.751378 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef0650e-42e0-4429-81f5-206d36e6bf5a None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:56.844233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:46:56.851701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef0650e-42e0-4429-81f5-206d36e6bf5a None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:56.884554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290c5723-7544-440d-95f6-85d4849235fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:56.885177 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290c5723-7544-440d-95f6-85d4849235fb None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:56.885456 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290c5723-7544-440d-95f6-85d4849235fb None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:56.885727 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290c5723-7544-440d-95f6-85d4849235fb None admin] PATH_INFO: `/v3/users/placement` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:56.892617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290c5723-7544-440d-95f6-85d4849235fb None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:56.894651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290c5723-7544-440d-95f6-85d4849235fb None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:56.914569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-290c5723-7544-440d-95f6-85d4849235fb None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 23 13:46:56.914569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:46:56.914569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:46:56.914569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:46:56.914569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:46:56.914569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:46:56.914569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:46:56.914569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:46:56.914569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:46:56.914569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:46:56.914569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:46:56.914569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:46:56.914569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:46:56.914569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:46:56.914569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:46:56.914569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:46:56.915725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:46:56.915725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:46:56.915725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:46:56.915725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:46:56.915725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:46:56.915725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:46:56.915725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:46:56.915725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:46:56.915725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:46:56.915725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:46:56.915725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:46:56.915725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:46:56.915725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:46:56.915725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:46:56.915725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:46:56.915725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:46:56.915725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:46:56.915725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:46:56.917017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:46:56.917017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:46:56.917017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:46:56.917017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:46:56.917017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:46:56.917017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:46:56.917017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:46:56.917017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:46:56.917017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 23 13:46:56.917017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:46:56.917017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 617/1226] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:46:56 2022] GET /identity/v3/users/placement => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:46:56.921605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3e1efb-76fb-4715-801b-3688278f576d None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:56.966952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:46:57.019710 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3e1efb-76fb-4715-801b-3688278f576d None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:57.054233 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de768b0d-83fd-4dcc-b0f7-8f432f0b52cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:57.054854 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de768b0d-83fd-4dcc-b0f7-8f432f0b52cc None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:57.055128 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de768b0d-83fd-4dcc-b0f7-8f432f0b52cc None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:57.055397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de768b0d-83fd-4dcc-b0f7-8f432f0b52cc None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:57.056108 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de768b0d-83fd-4dcc-b0f7-8f432f0b52cc None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:46:57.056406 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de768b0d-83fd-4dcc-b0f7-8f432f0b52cc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:57.057849 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de768b0d-83fd-4dcc-b0f7-8f432f0b52cc None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:57.074562 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 610/1227] 10.222.0.62 () {56 vars in 1143 bytes} [Fri Sep 23 13:46:56 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:46:57.080421 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290c5723-7544-440d-95f6-85d4849235fb None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:57.182036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290c5723-7544-440d-95f6-85d4849235fb None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:57.216322 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af53ece1-319a-4813-bbda-f2c36350b646 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:57.217003 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af53ece1-319a-4813-bbda-f2c36350b646 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:57.217225 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af53ece1-319a-4813-bbda-f2c36350b646 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:57.217494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af53ece1-319a-4813-bbda-f2c36350b646 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:57.229069 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af53ece1-319a-4813-bbda-f2c36350b646 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:57.230548 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af53ece1-319a-4813-bbda-f2c36350b646 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:57.245406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-af53ece1-319a-4813-bbda-f2c36350b646 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:46:57.245406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:46:57.245406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:46:57.245406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:46:57.245406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:46:57.245406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:46:57.245406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:46:57.245406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:46:57.245406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:46:57.245406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:46:57.245406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:46:57.245406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:46:57.245406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:46:57.245406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:46:57.245406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:46:57.245406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:46:57.246632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:46:57.246632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:46:57.246632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:46:57.246632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:46:57.246632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:46:57.246632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:46:57.246632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:46:57.246632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:46:57.246632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:46:57.246632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:46:57.246632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:46:57.246632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:46:57.246632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:46:57.246632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:46:57.246632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:46:57.246632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:46:57.246632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:46:57.247744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:46:57.247744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:46:57.247744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:46:57.247744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 618/1228] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:46:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:46:57.252434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de768b0d-83fd-4dcc-b0f7-8f432f0b52cc None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:57.351490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de768b0d-83fd-4dcc-b0f7-8f432f0b52cc None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:57.353820 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:46:57.385798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb7d7d9-fae9-4e38-83d2-2541b5396b3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:57.386419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb7d7d9-fae9-4e38-83d2-2541b5396b3f None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:57.386699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb7d7d9-fae9-4e38-83d2-2541b5396b3f None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:57.386968 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb7d7d9-fae9-4e38-83d2-2541b5396b3f None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:57.387596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb7d7d9-fae9-4e38-83d2-2541b5396b3f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:46:57.387895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb7d7d9-fae9-4e38-83d2-2541b5396b3f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:57.389244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb7d7d9-fae9-4e38-83d2-2541b5396b3f None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:57.400439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 611/1229] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:46:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:46:57.406242 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af53ece1-319a-4813-bbda-f2c36350b646 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:57.416277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:46:57.508878 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af53ece1-319a-4813-bbda-f2c36350b646 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:57.542926 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38635c2-3948-4eb5-8e26-05b5aec319e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:57.543539 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38635c2-3948-4eb5-8e26-05b5aec319e8 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:57.543817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38635c2-3948-4eb5-8e26-05b5aec319e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:57.544080 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38635c2-3948-4eb5-8e26-05b5aec319e8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:57.550857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38635c2-3948-4eb5-8e26-05b5aec319e8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:57.552289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38635c2-3948-4eb5-8e26-05b5aec319e8 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:57.563277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-a38635c2-3948-4eb5-8e26-05b5aec319e8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:46:57.563277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:46:57.563277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:46:57.563277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:46:57.563277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:46:57.563277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:46:57.563277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:46:57.563277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:46:57.563277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:46:57.563277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:46:57.563277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:46:57.563277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:46:57.563277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:46:57.563277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:46:57.563277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:46:57.564357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:46:57.564357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:46:57.564357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:46:57.564357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:46:57.564357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:46:57.564357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:46:57.564357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:46:57.564357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:46:57.564357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:46:57.564357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:46:57.564357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:46:57.564357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:46:57.564357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:46:57.564357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:46:57.564357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:46:57.564357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:46:57.564357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:46:57.565598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:46:57.565598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:46:57.565598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:46:57.565598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:46:57.565598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:46:57.565598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:46:57.565598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:46:57.565598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:46:57.565598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 619/1230] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:46:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:46:57.570366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb7d7d9-fae9-4e38-83d2-2541b5396b3f None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:57.673339 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb7d7d9-fae9-4e38-83d2-2541b5396b3f None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:57.705343 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af548139-ffeb-4f52-8b0b-35cdc48f4208 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:57.706511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af548139-ffeb-4f52-8b0b-35cdc48f4208 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:57.706786 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af548139-ffeb-4f52-8b0b-35cdc48f4208 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:57.707055 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af548139-ffeb-4f52-8b0b-35cdc48f4208 None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:57.707735 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af548139-ffeb-4f52-8b0b-35cdc48f4208 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:46:57.708038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af548139-ffeb-4f52-8b0b-35cdc48f4208 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:57.709470 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af548139-ffeb-4f52-8b0b-35cdc48f4208 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:57.721295 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 612/1231] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:46:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:46:57.728094 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38635c2-3948-4eb5-8e26-05b5aec319e8 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:46:57.793933 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:46:57.827516 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38635c2-3948-4eb5-8e26-05b5aec319e8 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:46:57.862303 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04034d2-26ed-4c40-94cf-c73a3ecf5dce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:46:57.863134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04034d2-26ed-4c40-94cf-c73a3ecf5dce None admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:46:57.863412 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04034d2-26ed-4c40-94cf-c73a3ecf5dce None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:46:57.863688 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04034d2-26ed-4c40-94cf-c73a3ecf5dce None admin] PATH_INFO: `/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/ce249d4a99c34891b6ba941f57a14dd4/roles/809612fc6d62442cbcc179b5c4123465` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:46:57.878720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04034d2-26ed-4c40-94cf-c73a3ecf5dce None admin] RBAC: Authorizing `identity:create_grant(project_id=2a19fd4315ad4a6eb349dfe8cfa66c23, user_id=ce249d4a99c34891b6ba941f57a14dd4, role_id=809612fc6d62442cbcc179b5c4123465)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:46:57.880158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04034d2-26ed-4c40-94cf-c73a3ecf5dce None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:46:57.949700 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 620/1232] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:46:57 2022] PUT /identity/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/ce249d4a99c34891b6ba941f57a14dd4/roles/809612fc6d62442cbcc179b5c4123465 => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 13:47:00.108322 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb04ed72-ff7d-4acf-8190-c770d4fd4646 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:00.108852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb04ed72-ff7d-4acf-8190-c770d4fd4646 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:00.108944 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb04ed72-ff7d-4acf-8190-c770d4fd4646 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:00.110248 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 613/1233] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:47:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:47:00.116115 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d9f914-9f8a-4f6e-99b7-bf134dea9ad8 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:00.116529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d9f914-9f8a-4f6e-99b7-bf134dea9ad8 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:00.116649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d9f914-9f8a-4f6e-99b7-bf134dea9ad8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:00.134744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-38d9f914-9f8a-4f6e-99b7-bf134dea9ad8 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:47:00.135954 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-38d9f914-9f8a-4f6e-99b7-bf134dea9ad8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:47:00.201782 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 621/1234] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:47:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:47:00.478091 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb04ed72-ff7d-4acf-8190-c770d4fd4646 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:00.502703 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:00.575823 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb04ed72-ff7d-4acf-8190-c770d4fd4646 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:00.609072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5cc634-26cf-4581-b337-02950eb905cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:00.609686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5cc634-26cf-4581-b337-02950eb905cd None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:00.609962 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5cc634-26cf-4581-b337-02950eb905cd None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:00.610231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5cc634-26cf-4581-b337-02950eb905cd None admin] PATH_INFO: `/v3/roles/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:00.619922 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a5cc634-26cf-4581-b337-02950eb905cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:47:00.624726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-1a5cc634-26cf-4581-b337-02950eb905cd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:47:00.624726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:00.624726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:00.624726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:00.624726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:00.624726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:00.624726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:00.624726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:00.624726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:00.624726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:00.624726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:00.624726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:00.624726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:47:00.624726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:47:00.624726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:47:00.624726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:47:00.625815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:47:00.625815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:47:00.625815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:47:00.625815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:47:00.625815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:00.625815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:00.625815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:00.625815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:00.625815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:00.625815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:00.625815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:00.625815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:00.625815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:00.625815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:00.625815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:00.625815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:00.626699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:00.626699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:00.626699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:00.626699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:00.626699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:47:00.626699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:47:00.626699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:47:00.626699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:47:00.626699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:47:00.626699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:47:00.626699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 23 13:47:00.626699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:47:00.626699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 614/1235] 10.222.0.62 () {56 vars in 1102 bytes} [Fri Sep 23 13:47:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:00.631890 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d9f914-9f8a-4f6e-99b7-bf134dea9ad8 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:00.679978 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:00.731506 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d9f914-9f8a-4f6e-99b7-bf134dea9ad8 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:00.763779 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ff895e-f42c-4a4a-bd86-9bb73308e308 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:00.764374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ff895e-f42c-4a4a-bd86-9bb73308e308 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:00.764653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ff895e-f42c-4a4a-bd86-9bb73308e308 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:00.764922 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ff895e-f42c-4a4a-bd86-9bb73308e308 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:00.765573 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ff895e-f42c-4a4a-bd86-9bb73308e308 None admin] RBAC: Adding query filter params (name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:00.765873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ff895e-f42c-4a4a-bd86-9bb73308e308 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:00.767297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4ff895e-f42c-4a4a-bd86-9bb73308e308 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:00.775693 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 622/1236] 10.222.0.62 () {56 vars in 1103 bytes} [Fri Sep 23 13:47:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:00.781549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5cc634-26cf-4581-b337-02950eb905cd None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:00.865691 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:00.881300 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5cc634-26cf-4581-b337-02950eb905cd None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:00.913806 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e495e446-7ee4-41ca-b65c-969b7bdb80b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:00.914440 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e495e446-7ee4-41ca-b65c-969b7bdb80b4 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:00.914724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e495e446-7ee4-41ca-b65c-969b7bdb80b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:00.914994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e495e446-7ee4-41ca-b65c-969b7bdb80b4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:00.925490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e495e446-7ee4-41ca-b65c-969b7bdb80b4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:00.926953 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e495e446-7ee4-41ca-b65c-969b7bdb80b4 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:00.941038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-e495e446-7ee4-41ca-b65c-969b7bdb80b4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:47:00.941038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:00.941038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:00.941038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:00.941038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:00.941038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:00.941038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:00.941038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:00.941038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:00.941038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:00.941038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:00.941038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:00.941038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:47:00.941038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:47:00.941038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:47:00.941038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:47:00.942121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:00.942121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:00.942121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:00.942121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:00.942121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:00.942121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:00.942121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:00.942121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:00.942121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:00.942121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:00.942121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:00.942121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:00.942121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:00.942121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:00.942121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:00.942121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:00.942121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:47:00.943616 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:47:00.943616 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:47:00.943616 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:47:00.943616 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 615/1237] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:47:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:00.947919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ff895e-f42c-4a4a-bd86-9bb73308e308 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:01.046198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ff895e-f42c-4a4a-bd86-9bb73308e308 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:01.055389 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:01.078800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1a4475-7968-4309-bd35-bd1712cb3bf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:01.079400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1a4475-7968-4309-bd35-bd1712cb3bf3 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:01.079673 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1a4475-7968-4309-bd35-bd1712cb3bf3 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:01.079934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1a4475-7968-4309-bd35-bd1712cb3bf3 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:01.080737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1a4475-7968-4309-bd35-bd1712cb3bf3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:01.081013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1a4475-7968-4309-bd35-bd1712cb3bf3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:01.082378 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1a4475-7968-4309-bd35-bd1712cb3bf3 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:01.093697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 623/1238] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:47:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:01.100332 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e495e446-7ee4-41ca-b65c-969b7bdb80b4 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:01.198897 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e495e446-7ee4-41ca-b65c-969b7bdb80b4 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:01.233384 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d52761-4162-47fa-8e96-2bb33e924529 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:01.234005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d52761-4162-47fa-8e96-2bb33e924529 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:01.234285 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d52761-4162-47fa-8e96-2bb33e924529 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:01.234594 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d52761-4162-47fa-8e96-2bb33e924529 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:01.237436 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:01.242564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d52761-4162-47fa-8e96-2bb33e924529 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:01.243974 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d52761-4162-47fa-8e96-2bb33e924529 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:01.254719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-90d52761-4162-47fa-8e96-2bb33e924529 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 23 13:47:01.254719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:01.254719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:01.254719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:01.254719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:01.254719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:01.254719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:01.254719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:01.254719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:01.254719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:01.254719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:01.254719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:01.254719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:47:01.254719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:47:01.254719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:47:01.254719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:47:01.255714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:01.255714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:01.255714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:47:01.255714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:47:01.255714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:47:01.255714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:47:01.255714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:01.255714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:01.255714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:01.255714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:01.255714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:01.255714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:01.255714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:01.255714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:01.255714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:01.255714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:01.255714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:01.255714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:01.256908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:01.256908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:01.256908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:47:01.256908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:47:01.256908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:47:01.256908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:47:01.256908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:47:01.256908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:47:01.256908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 23 13:47:01.256908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:47:01.256908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 616/1239] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:47:01 2022] GET /identity/v3/users/placement => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:01.261864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1a4475-7968-4309-bd35-bd1712cb3bf3 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:01.360981 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1a4475-7968-4309-bd35-bd1712cb3bf3 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:01.395759 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a9847f-dfbf-45b1-a2c3-a49520170fef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:01.396352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a9847f-dfbf-45b1-a2c3-a49520170fef None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:01.396626 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a9847f-dfbf-45b1-a2c3-a49520170fef None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:01.396893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a9847f-dfbf-45b1-a2c3-a49520170fef None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:01.397608 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a9847f-dfbf-45b1-a2c3-a49520170fef None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:01.397905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a9847f-dfbf-45b1-a2c3-a49520170fef None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:01.399332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a9847f-dfbf-45b1-a2c3-a49520170fef None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:01.416604 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 624/1240] 10.222.0.62 () {56 vars in 1143 bytes} [Fri Sep 23 13:47:01 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:01.422588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d52761-4162-47fa-8e96-2bb33e924529 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:01.523045 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d52761-4162-47fa-8e96-2bb33e924529 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:01.555348 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e341bc0-a5c2-4d6e-8a3d-b5d6e6f6dfcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:01.555947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e341bc0-a5c2-4d6e-8a3d-b5d6e6f6dfcb None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:01.556227 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e341bc0-a5c2-4d6e-8a3d-b5d6e6f6dfcb None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:01.556497 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e341bc0-a5c2-4d6e-8a3d-b5d6e6f6dfcb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:01.616335 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:01.622628 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e341bc0-a5c2-4d6e-8a3d-b5d6e6f6dfcb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:01.624314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e341bc0-a5c2-4d6e-8a3d-b5d6e6f6dfcb None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:01.719711 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 617/1241] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:47:01 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 23 13:47:01.725920 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a9847f-dfbf-45b1-a2c3-a49520170fef None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:01.752234 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:01.824669 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a9847f-dfbf-45b1-a2c3-a49520170fef None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:01.858069 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f439625a-19f2-42c3-b4c9-6534e2b41c62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:01.858716 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f439625a-19f2-42c3-b4c9-6534e2b41c62 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:01.858998 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f439625a-19f2-42c3-b4c9-6534e2b41c62 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:01.859270 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f439625a-19f2-42c3-b4c9-6534e2b41c62 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:01.873735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f439625a-19f2-42c3-b4c9-6534e2b41c62 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:01.873735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f439625a-19f2-42c3-b4c9-6534e2b41c62 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:01.886257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-f439625a-19f2-42c3-b4c9-6534e2b41c62 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:47:01.886257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:01.886257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:01.886257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:01.886257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:01.886257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:01.886257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:01.886257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:01.886257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:01.886257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:01.886257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:01.886257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:01.886257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:47:01.886257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:47:01.886257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:47:01.886257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:47:01.887419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:01.887419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:01.887419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:01.887419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:01.887419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:01.887419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:01.887419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:01.887419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:01.887419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:01.887419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:01.887419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:01.887419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:01.887419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:01.887419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:01.887419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:01.887419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:01.887419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:47:01.888582 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:47:01.888582 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:47:01.888582 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:47:01.888582 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 625/1242] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:47:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:01.893852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e341bc0-a5c2-4d6e-8a3d-b5d6e6f6dfcb None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:01.986533 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:01.992767 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e341bc0-a5c2-4d6e-8a3d-b5d6e6f6dfcb None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:02.025084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49dab0a-59f9-4403-b3fe-6b1b479e8f0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:02.025676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49dab0a-59f9-4403-b3fe-6b1b479e8f0a None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:02.025951 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49dab0a-59f9-4403-b3fe-6b1b479e8f0a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:02.026231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49dab0a-59f9-4403-b3fe-6b1b479e8f0a None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:02.026880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49dab0a-59f9-4403-b3fe-6b1b479e8f0a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:02.027177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49dab0a-59f9-4403-b3fe-6b1b479e8f0a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:02.028545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49dab0a-59f9-4403-b3fe-6b1b479e8f0a None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:02.039170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 618/1243] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:47:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:02.044920 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f439625a-19f2-42c3-b4c9-6534e2b41c62 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:02.124509 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:02.145576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f439625a-19f2-42c3-b4c9-6534e2b41c62 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:02.178438 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3327d86-44d8-44ce-a875-e6c3e1b7d61e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:02.179045 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3327d86-44d8-44ce-a875-e6c3e1b7d61e None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:02.179319 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3327d86-44d8-44ce-a875-e6c3e1b7d61e None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:02.179587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3327d86-44d8-44ce-a875-e6c3e1b7d61e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:02.185937 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3327d86-44d8-44ce-a875-e6c3e1b7d61e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:02.187389 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3327d86-44d8-44ce-a875-e6c3e1b7d61e None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:02.198994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-f3327d86-44d8-44ce-a875-e6c3e1b7d61e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:47:02.198994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:02.198994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:02.198994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:02.198994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:02.198994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:02.198994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:02.198994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:02.198994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:02.198994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:02.198994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:02.198994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:02.198994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:47:02.198994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:47:02.198994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:47:02.200197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:47:02.200197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:02.200197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:02.200197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:02.200197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:02.200197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:02.200197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:02.200197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:02.200197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:02.200197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:02.200197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:02.200197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:02.200197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:02.200197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:02.200197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:02.200197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:02.200197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:02.201293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:47:02.201293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:47:02.201293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:47:02.201293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:47:02.201293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:47:02.201293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:47:02.201293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:47:02.201293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:47:02.201293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 626/1244] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:47:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:02.206157 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49dab0a-59f9-4403-b3fe-6b1b479e8f0a None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:02.304717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49dab0a-59f9-4403-b3fe-6b1b479e8f0a None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:02.337425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934a3bc9-506a-47d3-a20c-f88e9173c938 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:02.338024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934a3bc9-506a-47d3-a20c-f88e9173c938 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:02.338300 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934a3bc9-506a-47d3-a20c-f88e9173c938 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:02.338616 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934a3bc9-506a-47d3-a20c-f88e9173c938 None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:02.339294 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934a3bc9-506a-47d3-a20c-f88e9173c938 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:02.339611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934a3bc9-506a-47d3-a20c-f88e9173c938 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:02.341005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934a3bc9-506a-47d3-a20c-f88e9173c938 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:02.355467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 619/1245] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:47:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 13:47:02.361527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3327d86-44d8-44ce-a875-e6c3e1b7d61e None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:02.460501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3327d86-44d8-44ce-a875-e6c3e1b7d61e None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:02.494106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d39f792-bcca-4bbf-9ebb-5c07ada261be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:02.494720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d39f792-bcca-4bbf-9ebb-5c07ada261be None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:02.494999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d39f792-bcca-4bbf-9ebb-5c07ada261be None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:02.495267 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d39f792-bcca-4bbf-9ebb-5c07ada261be None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:02.495980 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d39f792-bcca-4bbf-9ebb-5c07ada261be None admin] RBAC: Adding query filter params (role.id=809612fc6d62442cbcc179b5c4123465, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=ce249d4a99c34891b6ba941f57a14dd4) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:02.496279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d39f792-bcca-4bbf-9ebb-5c07ada261be None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:02.497741 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d39f792-bcca-4bbf-9ebb-5c07ada261be None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:02.503746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 627/1246] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:47:02 2022] GET /identity/v3/role_assignments?user.id=ce249d4a99c34891b6ba941f57a14dd4&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=809612fc6d62442cbcc179b5c4123465 => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:04.719587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca57179-9243-4815-9130-d83e31d14745 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:04.720100 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca57179-9243-4815-9130-d83e31d14745 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:04.720178 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca57179-9243-4815-9130-d83e31d14745 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:04.722262 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 620/1247] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:47:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:47:04.727780 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e0f14f-0a75-432f-ab39-411734295fb8 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:04.728183 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e0f14f-0a75-432f-ab39-411734295fb8 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:04.728396 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e0f14f-0a75-432f-ab39-411734295fb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:04.734715 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:04.748536 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-14e0f14f-0a75-432f-ab39-411734295fb8 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:47:04.749776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-14e0f14f-0a75-432f-ab39-411734295fb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:47:04.826511 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 628/1248] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:47:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:47:05.091600 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca57179-9243-4815-9130-d83e31d14745 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:05.110546 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:05.190992 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca57179-9243-4815-9130-d83e31d14745 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:05.225883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3567e902-6a89-42af-885d-5660affe325b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:05.226549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3567e902-6a89-42af-885d-5660affe325b None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:05.226994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3567e902-6a89-42af-885d-5660affe325b None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:05.227091 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3567e902-6a89-42af-885d-5660affe325b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:05.236855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3567e902-6a89-42af-885d-5660affe325b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:47:05.241870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-3567e902-6a89-42af-885d-5660affe325b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:47:05.241870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:05.241870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:05.241870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:05.241870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:05.241870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:05.241870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:05.241870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:05.241870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:05.241870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:05.241870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:05.241870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:05.241870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:47:05.241870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:47:05.241870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:47:05.241870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:47:05.242976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:47:05.242976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:47:05.242976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:47:05.242976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:47:05.242976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:05.242976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:05.242976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:05.242976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:05.242976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:05.242976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:05.242976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:05.242976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:05.242976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:05.242976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:05.242976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:05.242976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:05.247442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:05.247442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:05.247442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:05.247442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:05.247442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:47:05.247442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:47:05.247442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:47:05.247442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:47:05.247442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:47:05.247442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:47:05.247442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:47:05.247442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:47:05.247442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 621/1249] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:47:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:47:05.249020 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e0f14f-0a75-432f-ab39-411734295fb8 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:05.353116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e0f14f-0a75-432f-ab39-411734295fb8 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:05.383237 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:05.388323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffed3e9d-649f-4cb5-b740-693603b986e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:05.388790 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffed3e9d-649f-4cb5-b740-693603b986e0 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:05.389064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffed3e9d-649f-4cb5-b740-693603b986e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:05.389332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffed3e9d-649f-4cb5-b740-693603b986e0 None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:05.389981 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffed3e9d-649f-4cb5-b740-693603b986e0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:05.390287 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffed3e9d-649f-4cb5-b740-693603b986e0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:05.391677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffed3e9d-649f-4cb5-b740-693603b986e0 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:05.400232 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 629/1250] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:47:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:05.406286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3567e902-6a89-42af-885d-5660affe325b None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:05.485937 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:05.507924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3567e902-6a89-42af-885d-5660affe325b None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:05.540215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b54626-42dc-4e45-bd14-d58b36e7fa0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:05.540840 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b54626-42dc-4e45-bd14-d58b36e7fa0d None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:05.541114 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b54626-42dc-4e45-bd14-d58b36e7fa0d None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:05.541382 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b54626-42dc-4e45-bd14-d58b36e7fa0d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:05.551647 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b54626-42dc-4e45-bd14-d58b36e7fa0d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:05.553068 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b54626-42dc-4e45-bd14-d58b36e7fa0d None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:05.567830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-21b54626-42dc-4e45-bd14-d58b36e7fa0d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:47:05.567830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:05.567830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:05.567830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:05.567830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:05.567830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:05.567830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:05.567830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:05.567830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:05.567830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:05.567830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:05.567830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:05.567830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:47:05.567830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:47:05.567830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:47:05.567830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:47:05.571015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:05.571015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:05.571015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:05.571015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:05.571015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:05.571015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:05.571015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:05.571015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:05.571015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:05.571015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:05.571015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:05.571015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:05.571015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:05.571015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:05.571015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:05.571015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:05.571015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:47:05.571892 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:47:05.571892 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:47:05.571892 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:47:05.571892 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 622/1251] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:47:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:05.574705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffed3e9d-649f-4cb5-b740-693603b986e0 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:05.676721 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffed3e9d-649f-4cb5-b740-693603b986e0 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:05.709137 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6956b918-9980-42be-bda7-00795ea11180 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:05.709768 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6956b918-9980-42be-bda7-00795ea11180 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:05.710043 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6956b918-9980-42be-bda7-00795ea11180 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:05.710387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6956b918-9980-42be-bda7-00795ea11180 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:05.711020 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6956b918-9980-42be-bda7-00795ea11180 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:05.711313 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6956b918-9980-42be-bda7-00795ea11180 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:05.712651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6956b918-9980-42be-bda7-00795ea11180 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:05.724409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 630/1252] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:47:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:05.730240 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b54626-42dc-4e45-bd14-d58b36e7fa0d None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:05.828338 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b54626-42dc-4e45-bd14-d58b36e7fa0d None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:05.853780 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:05.861194 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef09888c-d3e5-4d5a-9a2f-01defa1c67a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:05.861795 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef09888c-d3e5-4d5a-9a2f-01defa1c67a4 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:05.862742 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef09888c-d3e5-4d5a-9a2f-01defa1c67a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:05.863111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef09888c-d3e5-4d5a-9a2f-01defa1c67a4 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:05.870302 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef09888c-d3e5-4d5a-9a2f-01defa1c67a4 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:05.871744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef09888c-d3e5-4d5a-9a2f-01defa1c67a4 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:05.883075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-ef09888c-d3e5-4d5a-9a2f-01defa1c67a4 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 23 13:47:05.883075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:05.883075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:05.883075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:05.883075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:05.883075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:05.883075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:05.883075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:05.883075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:05.883075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:05.883075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:05.883075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:05.883075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:47:05.883075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:47:05.883075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:47:05.883075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:47:05.884141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:05.884141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:05.884141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:47:05.884141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:47:05.884141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:47:05.884141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:47:05.884141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:05.884141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:05.884141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:05.884141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:05.884141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:05.884141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:05.884141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:05.884141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:05.884141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:05.884141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:05.884141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:05.884141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:05.885652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:05.885652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:05.885652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:47:05.885652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:47:05.885652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:47:05.885652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:47:05.885652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:47:05.885652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:47:05.885652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 23 13:47:05.885652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:47:05.885652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 623/1253] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:47:05 2022] GET /identity/v3/users/placement => generated 87 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:05.889946 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6956b918-9980-42be-bda7-00795ea11180 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:05.944750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:05.987826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6956b918-9980-42be-bda7-00795ea11180 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:06.022091 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20711954-a6a4-4e65-bae4-8307efe359eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:06.022725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20711954-a6a4-4e65-bae4-8307efe359eb None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:06.023002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20711954-a6a4-4e65-bae4-8307efe359eb None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:06.023267 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20711954-a6a4-4e65-bae4-8307efe359eb None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:06.023984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20711954-a6a4-4e65-bae4-8307efe359eb None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:06.024286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20711954-a6a4-4e65-bae4-8307efe359eb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:06.025694 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20711954-a6a4-4e65-bae4-8307efe359eb None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:06.043210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 631/1254] 10.222.0.62 () {56 vars in 1143 bytes} [Fri Sep 23 13:47:05 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:06.048987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef09888c-d3e5-4d5a-9a2f-01defa1c67a4 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:06.146085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef09888c-d3e5-4d5a-9a2f-01defa1c67a4 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:06.179450 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a26caa-a787-4310-b5d8-14ce705568b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:06.180037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a26caa-a787-4310-b5d8-14ce705568b4 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:06.180313 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a26caa-a787-4310-b5d8-14ce705568b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:06.180580 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a26caa-a787-4310-b5d8-14ce705568b4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:06.236376 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:06.248833 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a26caa-a787-4310-b5d8-14ce705568b4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:06.250257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a26caa-a787-4310-b5d8-14ce705568b4 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:06.348545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 624/1255] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:47:06 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 23 13:47:06.354446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20711954-a6a4-4e65-bae4-8307efe359eb None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:06.452365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20711954-a6a4-4e65-bae4-8307efe359eb None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:06.466128 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:06.487245 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e91f15-23c1-4944-a4c0-dface9b46c0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:06.487861 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e91f15-23c1-4944-a4c0-dface9b46c0d None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:06.488141 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e91f15-23c1-4944-a4c0-dface9b46c0d None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:06.488407 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e91f15-23c1-4944-a4c0-dface9b46c0d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:06.498081 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e91f15-23c1-4944-a4c0-dface9b46c0d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:06.499543 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e91f15-23c1-4944-a4c0-dface9b46c0d None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:06.514015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-c0e91f15-23c1-4944-a4c0-dface9b46c0d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:47:06.514015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:06.514015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:06.514015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:06.514015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:06.514015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:06.514015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:06.514015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:06.514015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:06.514015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:06.514015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:06.514015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:06.514015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:47:06.514015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:47:06.514015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:47:06.514015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:47:06.515171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:06.515171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:06.515171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:06.515171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:06.515171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:06.515171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:06.515171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:06.515171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:06.515171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:06.515171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:06.515171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:06.515171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:06.515171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:06.515171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:06.515171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:06.515171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:06.515171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:47:06.516206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:47:06.516206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:47:06.516206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:47:06.516206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 632/1256] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:47:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:06.521698 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a26caa-a787-4310-b5d8-14ce705568b4 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:06.606893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:06.618516 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a26caa-a787-4310-b5d8-14ce705568b4 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:06.651977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb28bd2-fbcf-4189-882a-482d13bca0f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:06.652574 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb28bd2-fbcf-4189-882a-482d13bca0f4 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:06.652849 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb28bd2-fbcf-4189-882a-482d13bca0f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:06.653113 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb28bd2-fbcf-4189-882a-482d13bca0f4 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:06.653746 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb28bd2-fbcf-4189-882a-482d13bca0f4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:06.654044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb28bd2-fbcf-4189-882a-482d13bca0f4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:06.655424 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb28bd2-fbcf-4189-882a-482d13bca0f4 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:06.670476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 625/1257] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:47:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:06.672684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e91f15-23c1-4944-a4c0-dface9b46c0d None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:06.771168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e91f15-23c1-4944-a4c0-dface9b46c0d None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:06.803293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bb5dd6-7766-4a83-8923-488a3022dac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:06.803904 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bb5dd6-7766-4a83-8923-488a3022dac5 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:06.804178 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bb5dd6-7766-4a83-8923-488a3022dac5 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:06.804444 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bb5dd6-7766-4a83-8923-488a3022dac5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:06.810858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8bb5dd6-7766-4a83-8923-488a3022dac5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:06.812266 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8bb5dd6-7766-4a83-8923-488a3022dac5 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:06.823793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-b8bb5dd6-7766-4a83-8923-488a3022dac5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:47:06.823793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:06.823793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:06.823793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:06.823793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:06.823793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:06.823793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:06.823793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:06.823793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:06.823793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:06.823793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:06.823793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:06.823793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:47:06.823793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:47:06.823793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:47:06.824873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:47:06.824873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:06.824873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:06.824873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:06.824873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:06.824873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:06.824873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:06.824873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:06.824873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:06.824873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:06.824873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:06.824873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:06.824873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:06.824873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:06.824873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:06.824873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:06.824873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:06.825901 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:47:06.825901 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:47:06.825901 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:47:06.825901 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:47:06.825901 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:47:06.825901 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:47:06.825901 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:47:06.825901 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:47:06.825901 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 633/1258] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:47:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:06.831570 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb28bd2-fbcf-4189-882a-482d13bca0f4 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:06.928288 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb28bd2-fbcf-4189-882a-482d13bca0f4 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:06.961784 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9926ab-27a3-4a9d-a7ce-d000914c5747 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:06.962509 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9926ab-27a3-4a9d-a7ce-d000914c5747 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:06.962831 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9926ab-27a3-4a9d-a7ce-d000914c5747 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:06.963100 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9926ab-27a3-4a9d-a7ce-d000914c5747 None admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:06.963894 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9926ab-27a3-4a9d-a7ce-d000914c5747 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:06.964196 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9926ab-27a3-4a9d-a7ce-d000914c5747 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:06.965745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9926ab-27a3-4a9d-a7ce-d000914c5747 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:06.977731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 626/1259] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:47:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:06.983845 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bb5dd6-7766-4a83-8923-488a3022dac5 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:07.017622 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:07.092523 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bb5dd6-7766-4a83-8923-488a3022dac5 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:07.126485 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51265ed8-df75-4256-ae8c-88fb740ca48b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:07.127070 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51265ed8-df75-4256-ae8c-88fb740ca48b None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:07.127850 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51265ed8-df75-4256-ae8c-88fb740ca48b None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:07.128470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51265ed8-df75-4256-ae8c-88fb740ca48b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:07.128951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51265ed8-df75-4256-ae8c-88fb740ca48b None admin] RBAC: Adding query filter params (role.id=727f0fa598e54b9a929d2cf6277c875e, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=ce249d4a99c34891b6ba941f57a14dd4) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:07.129251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51265ed8-df75-4256-ae8c-88fb740ca48b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:07.130684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51265ed8-df75-4256-ae8c-88fb740ca48b None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:07.135889 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 634/1260] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:47:06 2022] GET /identity/v3/role_assignments?user.id=ce249d4a99c34891b6ba941f57a14dd4&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=727f0fa598e54b9a929d2cf6277c875e => generated 259 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:09.283169 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c133a2dd-3447-471d-93e4-bdea95727904 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:09.284856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c133a2dd-3447-471d-93e4-bdea95727904 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:09.284856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c133a2dd-3447-471d-93e4-bdea95727904 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:09.285575 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 627/1261] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:47:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:47:09.290655 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be768e5-f07d-459d-bc15-4465440e6469 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:09.300651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be768e5-f07d-459d-bc15-4465440e6469 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:09.300651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be768e5-f07d-459d-bc15-4465440e6469 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:09.309404 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-8be768e5-f07d-459d-bc15-4465440e6469 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:47:09.310602 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8be768e5-f07d-459d-bc15-4465440e6469 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:47:09.376410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 635/1262] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:47:09 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:47:09.657474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c133a2dd-3447-471d-93e4-bdea95727904 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:09.668909 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:09.756303 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c133a2dd-3447-471d-93e4-bdea95727904 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:09.790500 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae401f8-b57e-4ca0-98e4-c46bb01018a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:09.791117 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae401f8-b57e-4ca0-98e4-c46bb01018a1 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:09.791393 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae401f8-b57e-4ca0-98e4-c46bb01018a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:09.791658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae401f8-b57e-4ca0-98e4-c46bb01018a1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:09.801946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae401f8-b57e-4ca0-98e4-c46bb01018a1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:47:09.806934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-6ae401f8-b57e-4ca0-98e4-c46bb01018a1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:47:09.806934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:09.806934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:09.806934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:09.806934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:09.806934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:09.806934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:09.806934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:09.806934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:09.806934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:09.806934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:09.806934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:09.806934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:47:09.806934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:47:09.806934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:47:09.806934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:47:09.808022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:47:09.808022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:47:09.808022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:47:09.808022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:47:09.808022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:09.808022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:09.808022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:09.808022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:09.808022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:09.808022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:09.808022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:09.808022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:09.808022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:09.808022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:09.808022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:09.808022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:09.808911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:09.808911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:09.808911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:09.808911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:09.808911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:47:09.808911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:47:09.808911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:47:09.808911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:47:09.808911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:47:09.808911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:47:09.808911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:47:09.808911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:47:09.808911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 628/1263] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:47:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:09.814183 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be768e5-f07d-459d-bc15-4465440e6469 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:09.833403 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:09.912109 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be768e5-f07d-459d-bc15-4465440e6469 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:09.945758 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce0e815-e62a-402f-bde1-95216e741f9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:09.946379 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce0e815-e62a-402f-bde1-95216e741f9a None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:09.946660 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce0e815-e62a-402f-bde1-95216e741f9a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:09.946928 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce0e815-e62a-402f-bde1-95216e741f9a None admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:09.947585 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce0e815-e62a-402f-bde1-95216e741f9a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:09.947882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce0e815-e62a-402f-bde1-95216e741f9a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:09.949234 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce0e815-e62a-402f-bde1-95216e741f9a None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:09.957624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 636/1264] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:47:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:09.963593 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae401f8-b57e-4ca0-98e4-c46bb01018a1 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:10.029041 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:10.061026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae401f8-b57e-4ca0-98e4-c46bb01018a1 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:10.093544 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd2b5e4-e288-4e0e-a502-158c81e3b007 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:10.094163 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd2b5e4-e288-4e0e-a502-158c81e3b007 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:10.094476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd2b5e4-e288-4e0e-a502-158c81e3b007 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:10.094746 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd2b5e4-e288-4e0e-a502-158c81e3b007 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:10.105284 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd2b5e4-e288-4e0e-a502-158c81e3b007 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:10.106725 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd2b5e4-e288-4e0e-a502-158c81e3b007 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:10.121394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-1bd2b5e4-e288-4e0e-a502-158c81e3b007 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:47:10.121394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:10.121394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:10.121394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:10.121394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:10.121394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:10.121394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:10.121394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:10.121394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:10.121394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:10.121394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:10.121394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:10.121394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:47:10.121394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:47:10.121394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:47:10.121394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:47:10.126686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:10.126686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:10.126686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:10.126686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:10.126686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:10.126686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:10.126686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:10.126686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:10.126686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:10.126686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:10.126686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:10.126686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:10.126686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:10.126686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:10.126686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:10.126686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:10.126686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:47:10.128820 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:47:10.128820 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:47:10.128820 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:47:10.128820 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 629/1265] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:47:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:10.129111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce0e815-e62a-402f-bde1-95216e741f9a None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:10.212290 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:10.230752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce0e815-e62a-402f-bde1-95216e741f9a None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:10.264435 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19d6840-afa7-4bcf-8e96-b6a8d2900ebc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:10.265033 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19d6840-afa7-4bcf-8e96-b6a8d2900ebc None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:10.265308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19d6840-afa7-4bcf-8e96-b6a8d2900ebc None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:10.265571 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19d6840-afa7-4bcf-8e96-b6a8d2900ebc None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:10.266207 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19d6840-afa7-4bcf-8e96-b6a8d2900ebc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:10.266535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19d6840-afa7-4bcf-8e96-b6a8d2900ebc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:10.268055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19d6840-afa7-4bcf-8e96-b6a8d2900ebc None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:10.278978 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 637/1266] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:47:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:10.285122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd2b5e4-e288-4e0e-a502-158c81e3b007 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:10.385592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd2b5e4-e288-4e0e-a502-158c81e3b007 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:10.404720 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:10.418421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2800ea8e-aa9c-4219-a301-4d9f74199127 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:10.419040 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2800ea8e-aa9c-4219-a301-4d9f74199127 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:10.419317 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2800ea8e-aa9c-4219-a301-4d9f74199127 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:10.419586 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2800ea8e-aa9c-4219-a301-4d9f74199127 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:10.427483 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2800ea8e-aa9c-4219-a301-4d9f74199127 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:10.428902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2800ea8e-aa9c-4219-a301-4d9f74199127 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:10.441002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-2800ea8e-aa9c-4219-a301-4d9f74199127 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 23 13:47:10.441002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:10.441002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:10.441002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:10.441002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:10.441002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:10.441002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:10.441002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:10.441002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:10.441002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:10.441002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:10.441002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:10.441002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:47:10.441002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:47:10.441002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:47:10.441002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:47:10.442095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:10.442095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:10.442095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:47:10.442095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:47:10.442095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:47:10.442095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:47:10.442095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:10.442095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:10.442095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:10.442095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:10.442095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:10.442095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:10.442095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:10.442095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:10.442095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:10.442095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:10.442095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:10.442095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:10.443277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:10.443277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:10.443277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:47:10.443277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:47:10.443277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:47:10.443277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:47:10.443277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:47:10.443277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:47:10.443277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 23 13:47:10.443277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:47:10.443277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 630/1267] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:47:10 2022] GET /identity/v3/users/placement => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:10.448115 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19d6840-afa7-4bcf-8e96-b6a8d2900ebc None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:10.550589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19d6840-afa7-4bcf-8e96-b6a8d2900ebc None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:10.584454 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcf0696-8c39-48fe-a074-e76c10637cce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:10.585074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcf0696-8c39-48fe-a074-e76c10637cce None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:10.585356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcf0696-8c39-48fe-a074-e76c10637cce None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:10.585627 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcf0696-8c39-48fe-a074-e76c10637cce None admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:10.586375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcf0696-8c39-48fe-a074-e76c10637cce None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:10.586681 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcf0696-8c39-48fe-a074-e76c10637cce None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:10.588199 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcf0696-8c39-48fe-a074-e76c10637cce None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:10.598707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:10.607276 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 638/1268] 10.222.0.62 () {56 vars in 1143 bytes} [Fri Sep 23 13:47:10 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:10.613843 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2800ea8e-aa9c-4219-a301-4d9f74199127 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:10.713728 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2800ea8e-aa9c-4219-a301-4d9f74199127 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:10.746804 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8488f8-4dac-4131-9201-d973f7d4342a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:10.747415 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8488f8-4dac-4131-9201-d973f7d4342a None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:10.747689 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8488f8-4dac-4131-9201-d973f7d4342a None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:10.747958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8488f8-4dac-4131-9201-d973f7d4342a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:10.757998 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8488f8-4dac-4131-9201-d973f7d4342a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:10.759461 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8488f8-4dac-4131-9201-d973f7d4342a None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:10.772987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-1c8488f8-4dac-4131-9201-d973f7d4342a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:47:10.772987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:10.772987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:10.772987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:10.772987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:10.772987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:10.772987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:10.772987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:10.772987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:10.772987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:10.772987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:10.772987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:10.772987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:47:10.772987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:47:10.772987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:47:10.772987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:47:10.778411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:10.778411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:10.778411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:10.778411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:10.778411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:10.778411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:10.778411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:10.778411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:10.778411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:10.778411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:10.778411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:10.778411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:10.778411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:10.778411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:10.778411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:10.778411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:10.778411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:47:10.779352 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:47:10.779352 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:47:10.779352 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:47:10.779352 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 631/1269] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:47:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:10.780858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcf0696-8c39-48fe-a074-e76c10637cce None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:10.881617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcf0696-8c39-48fe-a074-e76c10637cce None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:10.915556 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9967ee00-bcfc-4479-98ce-0c25e277e5e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:10.916155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9967ee00-bcfc-4479-98ce-0c25e277e5e1 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:10.916428 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9967ee00-bcfc-4479-98ce-0c25e277e5e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:10.916695 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9967ee00-bcfc-4479-98ce-0c25e277e5e1 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:10.917328 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9967ee00-bcfc-4479-98ce-0c25e277e5e1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:10.917623 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9967ee00-bcfc-4479-98ce-0c25e277e5e1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:10.919002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9967ee00-bcfc-4479-98ce-0c25e277e5e1 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:10.930042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 639/1270] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:47:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:10.936035 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8488f8-4dac-4131-9201-d973f7d4342a None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:10.967312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:11.034018 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8488f8-4dac-4131-9201-d973f7d4342a None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:11.066634 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b10092c-a0f3-4d36-a049-813169fc0cc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:11.067239 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b10092c-a0f3-4d36-a049-813169fc0cc7 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:11.067515 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b10092c-a0f3-4d36-a049-813169fc0cc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:11.067784 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b10092c-a0f3-4d36-a049-813169fc0cc7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:11.074109 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b10092c-a0f3-4d36-a049-813169fc0cc7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:11.075623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b10092c-a0f3-4d36-a049-813169fc0cc7 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:11.087016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-8b10092c-a0f3-4d36-a049-813169fc0cc7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:47:11.087016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:11.087016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:11.087016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:11.087016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:11.087016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:11.087016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:11.087016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:11.087016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:11.087016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:11.087016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:11.087016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:11.087016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:47:11.087016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:47:11.087016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:47:11.088057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:47:11.088057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:11.088057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:11.088057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:11.088057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:11.088057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:11.088057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:11.088057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:11.088057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:11.088057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:11.088057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:11.088057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:11.088057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:11.088057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:11.088057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:11.088057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:11.088057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:11.089054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:47:11.089054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:47:11.089054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:47:11.089054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:47:11.089054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:47:11.089054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:47:11.089054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:47:11.089054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:47:11.089054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 632/1271] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:47:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:11.094169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9967ee00-bcfc-4479-98ce-0c25e277e5e1 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:11.152771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:11.193952 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9967ee00-bcfc-4479-98ce-0c25e277e5e1 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:11.228202 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fa581a-0f01-4a89-8bd9-7701c7a89c59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:11.228803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fa581a-0f01-4a89-8bd9-7701c7a89c59 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:11.229078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fa581a-0f01-4a89-8bd9-7701c7a89c59 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:11.229347 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fa581a-0f01-4a89-8bd9-7701c7a89c59 None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:11.230030 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fa581a-0f01-4a89-8bd9-7701c7a89c59 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:11.230483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fa581a-0f01-4a89-8bd9-7701c7a89c59 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:11.231809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58fa581a-0f01-4a89-8bd9-7701c7a89c59 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:11.243451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 640/1272] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:47:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:11.249621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b10092c-a0f3-4d36-a049-813169fc0cc7 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:11.339500 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:11.348303 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b10092c-a0f3-4d36-a049-813169fc0cc7 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:11.381446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a90979e-0a4d-4a83-9090-10a20a65c775 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:11.382127 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a90979e-0a4d-4a83-9090-10a20a65c775 None admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:11.382431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a90979e-0a4d-4a83-9090-10a20a65c775 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:11.382699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a90979e-0a4d-4a83-9090-10a20a65c775 None admin] PATH_INFO: `/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/ce249d4a99c34891b6ba941f57a14dd4/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:11.384585 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a90979e-0a4d-4a83-9090-10a20a65c775 None admin] RBAC: Authorizing `identity:create_grant(project_id=2a19fd4315ad4a6eb349dfe8cfa66c23, user_id=ce249d4a99c34891b6ba941f57a14dd4, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:11.386032 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a90979e-0a4d-4a83-9090-10a20a65c775 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:11.468741 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 633/1273] 10.222.0.62 () {58 vars in 1439 bytes} [Fri Sep 23 13:47:11 2022] PUT /identity/v3/projects/2a19fd4315ad4a6eb349dfe8cfa66c23/users/ce249d4a99c34891b6ba941f57a14dd4/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:47:13.619783 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a75dd6-9c9f-4979-b39e-63a8dfd608b0 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:13.620285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a75dd6-9c9f-4979-b39e-63a8dfd608b0 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:13.620285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a75dd6-9c9f-4979-b39e-63a8dfd608b0 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:13.621579 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 641/1274] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:47:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:47:13.626771 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d38f4f6-bac3-4ba6-b7b1-e8f500ddc9a4 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:13.627208 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d38f4f6-bac3-4ba6-b7b1-e8f500ddc9a4 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:13.632388 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d38f4f6-bac3-4ba6-b7b1-e8f500ddc9a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:13.646618 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-9d38f4f6-bac3-4ba6-b7b1-e8f500ddc9a4 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:47:13.647809 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9d38f4f6-bac3-4ba6-b7b1-e8f500ddc9a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:47:13.714528 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 634/1275] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:47:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:47:13.986895 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a75dd6-9c9f-4979-b39e-63a8dfd608b0 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:14.096924 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a75dd6-9c9f-4979-b39e-63a8dfd608b0 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:14.116485 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:14.134342 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0d42a0-097a-472a-80d5-94d9e51c55fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:14.135001 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0d42a0-097a-472a-80d5-94d9e51c55fb None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:14.135277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0d42a0-097a-472a-80d5-94d9e51c55fb None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:14.135545 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0d42a0-097a-472a-80d5-94d9e51c55fb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:14.146638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0d42a0-097a-472a-80d5-94d9e51c55fb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 23 13:47:14.151475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-dc0d42a0-097a-472a-80d5-94d9e51c55fb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:47:14.151475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:14.151475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:14.151475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:14.151475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:14.151475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:14.151475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:14.151475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:14.151475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:14.151475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:14.151475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:14.151475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:14.151475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 23 13:47:14.151475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 23 13:47:14.151475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 23 13:47:14.151475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 23 13:47:14.152469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 23 13:47:14.152469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 23 13:47:14.152469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 23 13:47:14.152469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 23 13:47:14.152469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:14.152469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:14.152469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:14.152469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:14.152469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:14.152469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:14.152469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:14.152469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:14.152469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:14.152469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:14.152469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:14.152469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:14.153344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:14.153344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:14.153344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:14.153344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:14.153344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 23 13:47:14.153344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 23 13:47:14.153344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 23 13:47:14.153344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 23 13:47:14.153344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 23 13:47:14.153344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 23 13:47:14.153344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 23 13:47:14.153344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:47:14.153344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 642/1276] 10.222.0.62 () {56 vars in 1096 bytes} [Fri Sep 23 13:47:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:14.161143 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d38f4f6-bac3-4ba6-b7b1-e8f500ddc9a4 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:14.246135 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:14.265309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d38f4f6-bac3-4ba6-b7b1-e8f500ddc9a4 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:14.302264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cade1b09-d692-4eda-a981-b1c1eda6e7f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:14.302930 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cade1b09-d692-4eda-a981-b1c1eda6e7f5 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:14.303204 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cade1b09-d692-4eda-a981-b1c1eda6e7f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:14.303472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cade1b09-d692-4eda-a981-b1c1eda6e7f5 None admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:14.304142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cade1b09-d692-4eda-a981-b1c1eda6e7f5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:14.304449 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cade1b09-d692-4eda-a981-b1c1eda6e7f5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:14.305821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cade1b09-d692-4eda-a981-b1c1eda6e7f5 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:14.314228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 635/1277] 10.222.0.62 () {56 vars in 1099 bytes} [Fri Sep 23 13:47:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:14.320385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0d42a0-097a-472a-80d5-94d9e51c55fb None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:14.420021 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0d42a0-097a-472a-80d5-94d9e51c55fb None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:14.453237 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7838aec-57c8-4db7-8308-895cf0c7c7bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:14.453853 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7838aec-57c8-4db7-8308-895cf0c7c7bf None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:14.454128 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7838aec-57c8-4db7-8308-895cf0c7c7bf None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:14.454421 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7838aec-57c8-4db7-8308-895cf0c7c7bf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:14.464292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7838aec-57c8-4db7-8308-895cf0c7c7bf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:14.465710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7838aec-57c8-4db7-8308-895cf0c7c7bf None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:14.480657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-b7838aec-57c8-4db7-8308-895cf0c7c7bf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:47:14.480657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:14.480657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:14.480657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:14.480657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:14.480657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:14.480657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:14.480657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:14.480657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:14.480657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:14.480657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:14.480657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:14.480657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:47:14.480657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:47:14.480657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:47:14.480657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:47:14.481965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:14.481965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:14.481965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:14.481965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:14.481965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:14.481965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:14.481965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:14.481965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:14.481965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:14.481965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:14.481965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:14.481965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:14.481965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:14.481965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:14.481965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:14.481965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:14.481965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:47:14.483210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:47:14.483210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:47:14.483210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:47:14.483210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 643/1278] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:47:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:14.488260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cade1b09-d692-4eda-a981-b1c1eda6e7f5 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:14.589028 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cade1b09-d692-4eda-a981-b1c1eda6e7f5 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:14.623381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79db093d-fcc5-452b-a0e8-2f3083e7d588 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:14.623979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79db093d-fcc5-452b-a0e8-2f3083e7d588 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:14.624243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79db093d-fcc5-452b-a0e8-2f3083e7d588 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:14.624515 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79db093d-fcc5-452b-a0e8-2f3083e7d588 None admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:14.625195 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79db093d-fcc5-452b-a0e8-2f3083e7d588 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:14.625495 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79db093d-fcc5-452b-a0e8-2f3083e7d588 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:14.626909 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79db093d-fcc5-452b-a0e8-2f3083e7d588 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:14.638697 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 636/1279] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:47:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:14.644947 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7838aec-57c8-4db7-8308-895cf0c7c7bf None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:14.676001 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:14.744152 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7838aec-57c8-4db7-8308-895cf0c7c7bf None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:14.777179 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadf4213-13b5-4a1b-9ffb-a80271bba9a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:14.777797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadf4213-13b5-4a1b-9ffb-a80271bba9a2 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:14.778104 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadf4213-13b5-4a1b-9ffb-a80271bba9a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:14.778416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadf4213-13b5-4a1b-9ffb-a80271bba9a2 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:14.786001 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cadf4213-13b5-4a1b-9ffb-a80271bba9a2 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:14.787596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cadf4213-13b5-4a1b-9ffb-a80271bba9a2 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:14.799057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-cadf4213-13b5-4a1b-9ffb-a80271bba9a2 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 23 13:47:14.799057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:14.799057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:14.799057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:14.799057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:14.799057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:14.799057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:14.799057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:14.799057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:14.799057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:14.799057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:14.799057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:14.799057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 23 13:47:14.799057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 23 13:47:14.799057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 23 13:47:14.799057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 23 13:47:14.800214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:14.800214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:14.800214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 23 13:47:14.800214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:47:14.800214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 23 13:47:14.800214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 23 13:47:14.800214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:14.800214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:14.800214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:14.800214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:14.800214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:14.800214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:14.800214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:14.800214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:14.800214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:14.800214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:14.800214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:14.800214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:14.801286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:14.801286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:14.801286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 23 13:47:14.801286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 23 13:47:14.801286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 23 13:47:14.801286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 23 13:47:14.801286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 23 13:47:14.801286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 23 13:47:14.801286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 23 13:47:14.801286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:47:14.801286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 644/1280] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:47:14 2022] GET /identity/v3/users/placement => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:14.806269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79db093d-fcc5-452b-a0e8-2f3083e7d588 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:14.810124 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:14.903527 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79db093d-fcc5-452b-a0e8-2f3083e7d588 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:14.935991 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad8b4d8-873d-4308-ab36-34334023bce1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:14.936579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad8b4d8-873d-4308-ab36-34334023bce1 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:14.936853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad8b4d8-873d-4308-ab36-34334023bce1 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:14.937118 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ad8b4d8-873d-4308-ab36-34334023bce1 None admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:14.937825 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad8b4d8-873d-4308-ab36-34334023bce1 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:14.938123 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad8b4d8-873d-4308-ab36-34334023bce1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:14.939571 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ad8b4d8-873d-4308-ab36-34334023bce1 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:14.956681 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 637/1281] 10.222.0.62 () {56 vars in 1143 bytes} [Fri Sep 23 13:47:14 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:14.962432 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadf4213-13b5-4a1b-9ffb-a80271bba9a2 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:15.052532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:15.061759 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadf4213-13b5-4a1b-9ffb-a80271bba9a2 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:15.095218 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c5a0ac-bb14-44d0-8f74-822f5f83a19f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:15.095849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c5a0ac-bb14-44d0-8f74-822f5f83a19f None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:15.096479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c5a0ac-bb14-44d0-8f74-822f5f83a19f None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:15.096847 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c5a0ac-bb14-44d0-8f74-822f5f83a19f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:15.164353 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c5a0ac-bb14-44d0-8f74-822f5f83a19f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:15.165714 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c5a0ac-bb14-44d0-8f74-822f5f83a19f None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:15.253454 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:15.263433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 645/1282] 10.222.0.62 () {58 vars in 1282 bytes} [Fri Sep 23 13:47:14 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 23 13:47:15.270080 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad8b4d8-873d-4308-ab36-34334023bce1 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:15.334538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:15.372978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ad8b4d8-873d-4308-ab36-34334023bce1 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:15.406185 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8875e516-97ac-458e-a0df-dfde4a577eda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:15.406825 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8875e516-97ac-458e-a0df-dfde4a577eda None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:15.407098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8875e516-97ac-458e-a0df-dfde4a577eda None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:15.407363 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8875e516-97ac-458e-a0df-dfde4a577eda None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:15.419114 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8875e516-97ac-458e-a0df-dfde4a577eda None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:15.419114 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8875e516-97ac-458e-a0df-dfde4a577eda None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:15.433238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-8875e516-97ac-458e-a0df-dfde4a577eda None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:47:15.433238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:15.433238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:15.433238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:15.433238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:15.433238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:15.433238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:15.433238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:15.433238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:15.433238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:15.433238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:15.433238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:15.433238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 23 13:47:15.433238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 23 13:47:15.433238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 23 13:47:15.433238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 23 13:47:15.434202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:15.434202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:15.434202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:15.434202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:15.434202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:15.434202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:15.434202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:15.434202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:15.434202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:15.434202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:15.434202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:15.434202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:15.434202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:15.434202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:15.434202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:15.434202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:15.434202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 23 13:47:15.435257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 23 13:47:15.435257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 23 13:47:15.435257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:47:15.435257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 638/1283] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:47:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:15.441145 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c5a0ac-bb14-44d0-8f74-822f5f83a19f None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:15.538992 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c5a0ac-bb14-44d0-8f74-822f5f83a19f None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:15.572897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966bbe3b-3d64-44ec-a65e-82087b570510 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:15.573492 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966bbe3b-3d64-44ec-a65e-82087b570510 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:15.573798 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966bbe3b-3d64-44ec-a65e-82087b570510 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:15.574067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966bbe3b-3d64-44ec-a65e-82087b570510 None admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:15.574736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966bbe3b-3d64-44ec-a65e-82087b570510 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:15.575036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966bbe3b-3d64-44ec-a65e-82087b570510 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:15.576457 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966bbe3b-3d64-44ec-a65e-82087b570510 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:15.587373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 646/1284] 10.222.0.62 () {56 vars in 1109 bytes} [Fri Sep 23 13:47:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:15.593088 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8875e516-97ac-458e-a0df-dfde4a577eda None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:15.692475 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8875e516-97ac-458e-a0df-dfde4a577eda None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:15.705454 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:15.725179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b84863-9b69-488c-aacf-09c9e493ccfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:15.725795 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b84863-9b69-488c-aacf-09c9e493ccfb None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:15.726080 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b84863-9b69-488c-aacf-09c9e493ccfb None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:15.726359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b84863-9b69-488c-aacf-09c9e493ccfb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:15.732738 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b84863-9b69-488c-aacf-09c9e493ccfb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:15.734156 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b84863-9b69-488c-aacf-09c9e493ccfb None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:15.745717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application [None req-18b84863-9b69-488c-aacf-09c9e493ccfb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:47:15.745717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:15.745717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:15.745717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:15.745717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:15.745717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:15.745717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:15.745717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:15.745717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:15.745717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:15.745717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:15.745717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:15.745717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 23 13:47:15.745717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 23 13:47:15.745717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 23 13:47:15.746877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 23 13:47:15.746877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:15.746877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:15.746877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:15.746877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:15.746877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:15.746877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:15.746877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:15.746877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:15.746877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:15.746877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:15.746877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:15.746877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:15.746877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:15.746877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:15.746877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:15.746877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:15.747865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 23 13:47:15.747865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 23 13:47:15.747865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 23 13:47:15.747865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 23 13:47:15.747865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 23 13:47:15.747865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 23 13:47:15.747865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 23 13:47:15.747865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: ERROR keystone.server.flask.application Sep 23 13:47:15.747865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 639/1285] 10.222.0.62 () {56 vars in 1111 bytes} [Fri Sep 23 13:47:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:15.752810 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966bbe3b-3d64-44ec-a65e-82087b570510 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:15.802064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:15.851611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966bbe3b-3d64-44ec-a65e-82087b570510 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:15.884558 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cd6cb6-5fde-41ed-90be-1d3059b083d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:15.885160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cd6cb6-5fde-41ed-90be-1d3059b083d0 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:15.885435 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cd6cb6-5fde-41ed-90be-1d3059b083d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:15.885704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cd6cb6-5fde-41ed-90be-1d3059b083d0 None admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:15.886409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42cd6cb6-5fde-41ed-90be-1d3059b083d0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:15.886711 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42cd6cb6-5fde-41ed-90be-1d3059b083d0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:15.888120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42cd6cb6-5fde-41ed-90be-1d3059b083d0 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:15.900296 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 647/1286] 10.222.0.62 () {56 vars in 1148 bytes} [Fri Sep 23 13:47:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:15.906312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b84863-9b69-488c-aacf-09c9e493ccfb None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:16.005007 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b84863-9b69-488c-aacf-09c9e493ccfb None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:16.037156 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-381b9d7d-9ab4-4a3b-9919-df6435935471 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:16.037748 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381b9d7d-9ab4-4a3b-9919-df6435935471 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:16.038025 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381b9d7d-9ab4-4a3b-9919-df6435935471 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:16.038293 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-381b9d7d-9ab4-4a3b-9919-df6435935471 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:16.039030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381b9d7d-9ab4-4a3b-9919-df6435935471 None admin] RBAC: Adding query filter params (role.id=727f0fa598e54b9a929d2cf6277c875e, scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23, user.id=ce249d4a99c34891b6ba941f57a14dd4) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:16.039333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381b9d7d-9ab4-4a3b-9919-df6435935471 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:16.040735 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-381b9d7d-9ab4-4a3b-9919-df6435935471 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:16.046265 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 640/1287] 10.222.0.62 () {56 vars in 1374 bytes} [Fri Sep 23 13:47:15 2022] GET /identity/v3/role_assignments?user.id=ce249d4a99c34891b6ba941f57a14dd4&scope.project.id=2a19fd4315ad4a6eb349dfe8cfa66c23&role.id=727f0fa598e54b9a929d2cf6277c875e => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:18.228648 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01214ad6-9e36-4d60-8935-8148d6bf8b4c None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:18.229088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01214ad6-9e36-4d60-8935-8148d6bf8b4c None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:18.229180 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01214ad6-9e36-4d60-8935-8148d6bf8b4c None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:18.230736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 648/1288] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:47:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:47:18.236122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dd1e87-8c5d-4b89-a5db-719c799d5b87 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:18.236511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dd1e87-8c5d-4b89-a5db-719c799d5b87 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:18.236674 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dd1e87-8c5d-4b89-a5db-719c799d5b87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:18.255264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-c0dd1e87-8c5d-4b89-a5db-719c799d5b87 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:47:18.256555 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-c0dd1e87-8c5d-4b89-a5db-719c799d5b87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:47:18.285835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:18.323050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 641/1289] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:47:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:47:18.602888 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01214ad6-9e36-4d60-8935-8148d6bf8b4c None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:18.704953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01214ad6-9e36-4d60-8935-8148d6bf8b4c None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:18.716924 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:18.741437 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1beb07-80c8-405f-aac2-05a736be836a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:18.742046 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1beb07-80c8-405f-aac2-05a736be836a None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:18.742321 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1beb07-80c8-405f-aac2-05a736be836a None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:18.742616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1beb07-80c8-405f-aac2-05a736be836a None admin] PATH_INFO: `/v3/services/placement` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:18.743186 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1beb07-80c8-405f-aac2-05a736be836a None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:18.744542 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1beb07-80c8-405f-aac2-05a736be836a None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:18.753366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-ed1beb07-80c8-405f-aac2-05a736be836a None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 23 13:47:18.753366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:18.753366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:47:18.753366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:47:18.753366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:47:18.753366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:47:18.753366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:47:18.753366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:47:18.753366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 23 13:47:18.753366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:47:18.753366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:47:18.753366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:47:18.753366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:18.753366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:18.753366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:18.753366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:18.753366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:18.758964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:18.758964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:18.758964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:18.758964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:18.758964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:18.758964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:18.758964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:47:18.758964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:47:18.758964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:47:18.758964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:47:18.758964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:18.758964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:18.758964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:18.758964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:18.758964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:18.758964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:18.760120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:18.760120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:18.760120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:18.760120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:18.760120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:18.760120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:18.760120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:18.760120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:18.760120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:18.760120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:18.760120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:47:18.760120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:47:18.760120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 23 13:47:18.760120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:47:18.760120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 649/1290] 10.222.0.62 () {56 vars in 1117 bytes} [Fri Sep 23 13:47:18 2022] GET /identity/v3/services/placement => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:18.761006 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dd1e87-8c5d-4b89-a5db-719c799d5b87 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:18.855157 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dd1e87-8c5d-4b89-a5db-719c799d5b87 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:18.888082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30114ede-b45f-43bd-a7c7-6f759b563e79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:18.888666 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30114ede-b45f-43bd-a7c7-6f759b563e79 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:18.888939 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30114ede-b45f-43bd-a7c7-6f759b563e79 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:18.889202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30114ede-b45f-43bd-a7c7-6f759b563e79 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:18.889811 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30114ede-b45f-43bd-a7c7-6f759b563e79 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:18.890105 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30114ede-b45f-43bd-a7c7-6f759b563e79 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:18.891468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30114ede-b45f-43bd-a7c7-6f759b563e79 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:18.896267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 642/1291] 10.222.0.62 () {56 vars in 1116 bytes} [Fri Sep 23 13:47:18 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:18.901933 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1beb07-80c8-405f-aac2-05a736be836a None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:18.999959 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1beb07-80c8-405f-aac2-05a736be836a None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:19.032198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a93db04-a69f-4e07-9875-66d2e5fe509b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:19.035181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a93db04-a69f-4e07-9875-66d2e5fe509b None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:19.035181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a93db04-a69f-4e07-9875-66d2e5fe509b None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:19.035181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a93db04-a69f-4e07-9875-66d2e5fe509b None admin] PATH_INFO: `/v3/services` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:19.035181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a93db04-a69f-4e07-9875-66d2e5fe509b None admin] RBAC: Adding query filter params (type=placement) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:19.035181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a93db04-a69f-4e07-9875-66d2e5fe509b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:19.036293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a93db04-a69f-4e07-9875-66d2e5fe509b None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:19.040617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 650/1292] 10.222.0.62 () {56 vars in 1116 bytes} [Fri Sep 23 13:47:18 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:21.182904 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e9c140-30f9-46b8-a2eb-6e148a25d049 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:21.183571 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e9c140-30f9-46b8-a2eb-6e148a25d049 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:21.183571 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e9c140-30f9-46b8-a2eb-6e148a25d049 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:21.184769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 643/1293] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:47:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:47:21.190433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65b32e9-85f6-46ca-b3da-b9f2a26daa32 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:21.190833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65b32e9-85f6-46ca-b3da-b9f2a26daa32 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:21.191077 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65b32e9-85f6-46ca-b3da-b9f2a26daa32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:21.210012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-d65b32e9-85f6-46ca-b3da-b9f2a26daa32 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:47:21.211259 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d65b32e9-85f6-46ca-b3da-b9f2a26daa32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:47:21.241136 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:21.278250 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 651/1294] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:47:21 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:47:21.545715 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e9c140-30f9-46b8-a2eb-6e148a25d049 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:21.576714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:21.646585 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e9c140-30f9-46b8-a2eb-6e148a25d049 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:21.678853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43a592b-b1b6-465d-920f-2f499c59b20d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:21.679441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43a592b-b1b6-465d-920f-2f499c59b20d None admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:21.679715 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43a592b-b1b6-465d-920f-2f499c59b20d None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:21.679979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43a592b-b1b6-465d-920f-2f499c59b20d None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:21.680618 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43a592b-b1b6-465d-920f-2f499c59b20d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:21.681960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43a592b-b1b6-465d-920f-2f499c59b20d None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:21.771240 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 644/1295] 10.222.0.62 () {60 vars in 1141 bytes} [Fri Sep 23 13:47:21 2022] POST /identity/v3/services => generated 246 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:47:23.910940 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e0dfd6-a344-47a4-96a4-4e1f0ed5e23d None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:23.911491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e0dfd6-a344-47a4-96a4-4e1f0ed5e23d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:23.911491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e0dfd6-a344-47a4-96a4-4e1f0ed5e23d None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:23.912833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 652/1296] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:47:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:47:23.917970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edee7278-4188-4c5d-8207-5e2db7c9edc3 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:23.918383 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edee7278-4188-4c5d-8207-5e2db7c9edc3 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:23.918530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edee7278-4188-4c5d-8207-5e2db7c9edc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:23.936346 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-edee7278-4188-4c5d-8207-5e2db7c9edc3 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:47:23.937514 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-edee7278-4188-4c5d-8207-5e2db7c9edc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:47:24.009526 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 645/1297] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:47:23 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:47:24.278399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e0dfd6-a344-47a4-96a4-4e1f0ed5e23d None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:24.379647 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e0dfd6-a344-47a4-96a4-4e1f0ed5e23d None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:24.412111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa436c8-7bf7-4f9f-b726-3ee3a49790b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:24.412793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa436c8-7bf7-4f9f-b726-3ee3a49790b8 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:24.413070 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa436c8-7bf7-4f9f-b726-3ee3a49790b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:24.413335 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa436c8-7bf7-4f9f-b726-3ee3a49790b8 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:24.413904 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa436c8-7bf7-4f9f-b726-3ee3a49790b8 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:24.415358 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa436c8-7bf7-4f9f-b726-3ee3a49790b8 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:24.424308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-9fa436c8-7bf7-4f9f-b726-3ee3a49790b8 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 23 13:47:24.424308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:24.424308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:47:24.424308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:47:24.424308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:47:24.424308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:47:24.424308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:47:24.424308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:47:24.424308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 23 13:47:24.424308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:47:24.424308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:47:24.424308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:47:24.424308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:24.424308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:24.424308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:24.424308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:24.424308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:24.425434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:24.425434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:24.425434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:24.425434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:24.425434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:24.425434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:24.425434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:47:24.425434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:47:24.425434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:47:24.425434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:47:24.425434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:24.425434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:24.425434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:24.425434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:24.425434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:24.425434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:24.432568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:24.432568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:24.432568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:24.432568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:24.432568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:24.432568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:24.432568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:24.432568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:24.432568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:24.432568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:24.432568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:47:24.432568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:47:24.432568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 23 13:47:24.432568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:47:24.432568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 653/1298] 10.222.0.62 () {56 vars in 1117 bytes} [Fri Sep 23 13:47:24 2022] GET /identity/v3/services/placement => generated 90 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 23 13:47:24.433545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edee7278-4188-4c5d-8207-5e2db7c9edc3 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:24.450292 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:24.528799 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edee7278-4188-4c5d-8207-5e2db7c9edc3 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:24.563451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884a56a8-7315-4821-acfd-d8634a1652b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:24.564106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884a56a8-7315-4821-acfd-d8634a1652b2 None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:24.564952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884a56a8-7315-4821-acfd-d8634a1652b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:24.565224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884a56a8-7315-4821-acfd-d8634a1652b2 None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:24.565849 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884a56a8-7315-4821-acfd-d8634a1652b2 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:24.566259 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884a56a8-7315-4821-acfd-d8634a1652b2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:24.567642 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884a56a8-7315-4821-acfd-d8634a1652b2 None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:24.572662 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 646/1299] 10.222.0.62 () {56 vars in 1116 bytes} [Fri Sep 23 13:47:24 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:24.578892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa436c8-7bf7-4f9f-b726-3ee3a49790b8 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:24.610352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:24.678573 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa436c8-7bf7-4f9f-b726-3ee3a49790b8 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:24.711860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585005e6-7e58-4be0-ab52-69b8ac3e0037 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:24.712447 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585005e6-7e58-4be0-ab52-69b8ac3e0037 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:24.712726 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585005e6-7e58-4be0-ab52-69b8ac3e0037 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:24.712993 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585005e6-7e58-4be0-ab52-69b8ac3e0037 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:24.713689 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585005e6-7e58-4be0-ab52-69b8ac3e0037 None admin] RBAC: Adding query filter params (interface=public, service_id=e8c7852a254245e69448bbb113cc28aa, region_id=RegionOne) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:24.713992 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585005e6-7e58-4be0-ab52-69b8ac3e0037 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:24.715384 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585005e6-7e58-4be0-ab52-69b8ac3e0037 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:24.720242 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 654/1300] 10.222.0.62 () {56 vars in 1251 bytes} [Fri Sep 23 13:47:24 2022] GET /identity/v3/endpoints?service_id=e8c7852a254245e69448bbb113cc28aa&interface=public®ion_id=RegionOne => generated 194 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:24.727116 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884a56a8-7315-4821-acfd-d8634a1652b2 None admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:24.808499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:24.825889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884a56a8-7315-4821-acfd-d8634a1652b2 None admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:24.859939 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70786fd8-bb7d-4258-b8a3-db61303b65ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:24.860595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70786fd8-bb7d-4258-b8a3-db61303b65ef None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:24.860901 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70786fd8-bb7d-4258-b8a3-db61303b65ef None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:24.861742 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70786fd8-bb7d-4258-b8a3-db61303b65ef None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:24.862363 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70786fd8-bb7d-4258-b8a3-db61303b65ef None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:24.863723 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70786fd8-bb7d-4258-b8a3-db61303b65ef None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:24.870342 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 647/1301] 10.222.0.62 () {56 vars in 1087 bytes} [Fri Sep 23 13:47:24 2022] GET /identity/v3/services => generated 2191 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 13:47:27.087270 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6221f556-2fee-4870-a7d9-040f37224587 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:27.087829 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6221f556-2fee-4870-a7d9-040f37224587 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:27.087829 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6221f556-2fee-4870-a7d9-040f37224587 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:27.089180 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 655/1302] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:47:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:47:27.094527 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe1b88f-e666-4d57-b2c0-50d4652b652b None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:27.094942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe1b88f-e666-4d57-b2c0-50d4652b652b None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:27.095152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe1b88f-e666-4d57-b2c0-50d4652b652b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:27.114604 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-bbe1b88f-e666-4d57-b2c0-50d4652b652b None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:47:27.115777 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-bbe1b88f-e666-4d57-b2c0-50d4652b652b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:47:27.182692 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 648/1303] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:47:27 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 23 13:47:27.452618 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6221f556-2fee-4870-a7d9-040f37224587 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:27.546357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:27.553633 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6221f556-2fee-4870-a7d9-040f37224587 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:27.587371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014919bf-fbfb-49e1-aa5c-f398f815c7e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:27.587993 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014919bf-fbfb-49e1-aa5c-f398f815c7e2 None admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:27.588272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014919bf-fbfb-49e1-aa5c-f398f815c7e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:27.588541 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014919bf-fbfb-49e1-aa5c-f398f815c7e2 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:27.589115 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014919bf-fbfb-49e1-aa5c-f398f815c7e2 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:27.590497 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014919bf-fbfb-49e1-aa5c-f398f815c7e2 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:27.599416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application [None req-014919bf-fbfb-49e1-aa5c-f398f815c7e2 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 23 13:47:27.599416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:27.599416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 23 13:47:27.599416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 23 13:47:27.599416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 23 13:47:27.599416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 23 13:47:27.599416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 23 13:47:27.599416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:47:27.599416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 23 13:47:27.599416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:47:27.599416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 23 13:47:27.599416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:47:27.599416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 23 13:47:27.599416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 23 13:47:27.599416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 23 13:47:27.599416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 23 13:47:27.599416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 23 13:47:27.600502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 23 13:47:27.600502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 23 13:47:27.600502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 23 13:47:27.600502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 23 13:47:27.600502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 23 13:47:27.600502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 23 13:47:27.600502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 23 13:47:27.600502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 23 13:47:27.600502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 23 13:47:27.600502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 23 13:47:27.600502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 23 13:47:27.600502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 23 13:47:27.600502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 23 13:47:27.600502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 23 13:47:27.600502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 23 13:47:27.600502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.get_or_create( Sep 23 13:47:27.601349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 23 13:47:27.601349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application with Lock( Sep 23 13:47:27.601349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 23 13:47:27.601349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self._enter() Sep 23 13:47:27.601349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 23 13:47:27.601349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 23 13:47:27.601349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 23 13:47:27.601349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application return self.creator() Sep 23 13:47:27.601349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 23 13:47:27.601349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application created_value = creator( Sep 23 13:47:27.601349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 23 13:47:27.601349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 23 13:47:27.601349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 23 13:47:27.601349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: ERROR keystone.server.flask.application Sep 23 13:47:27.601349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 656/1304] 10.222.0.62 () {56 vars in 1117 bytes} [Fri Sep 23 13:47:27 2022] GET /identity/v3/services/placement => generated 90 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 23 13:47:27.607979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe1b88f-e666-4d57-b2c0-50d4652b652b None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:27.684814 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:27.706944 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe1b88f-e666-4d57-b2c0-50d4652b652b None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:27.740749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7d4a04-9f8b-471f-b02a-bc0da3d3cbdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:27.741336 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7d4a04-9f8b-471f-b02a-bc0da3d3cbdf None admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:27.741606 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7d4a04-9f8b-471f-b02a-bc0da3d3cbdf None admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:27.741870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7d4a04-9f8b-471f-b02a-bc0da3d3cbdf None admin] PATH_INFO: `/v3/services` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:27.742519 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7d4a04-9f8b-471f-b02a-bc0da3d3cbdf None admin] RBAC: Adding query filter params (name=placement) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:47:27.742814 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7d4a04-9f8b-471f-b02a-bc0da3d3cbdf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:27.744151 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7d4a04-9f8b-471f-b02a-bc0da3d3cbdf None admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:27.749077 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 649/1305] 10.222.0.62 () {56 vars in 1116 bytes} [Fri Sep 23 13:47:27 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:47:27.755185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014919bf-fbfb-49e1-aa5c-f398f815c7e2 None admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:47:27.858677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014919bf-fbfb-49e1-aa5c-f398f815c7e2 None admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:47:27.891514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6457e1-945d-4d8f-b34c-897dbf8b7be6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:47:27.892154 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6457e1-945d-4d8f-b34c-897dbf8b7be6 None admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:47:27.892426 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6457e1-945d-4d8f-b34c-897dbf8b7be6 None admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:47:27.892695 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6457e1-945d-4d8f-b34c-897dbf8b7be6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:47:27.893354 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6457e1-945d-4d8f-b34c-897dbf8b7be6 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:47:27.894734 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6457e1-945d-4d8f-b34c-897dbf8b7be6 None admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:47:28.034018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 657/1306] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 13:47:27 2022] POST /identity/v3/endpoints => generated 331 bytes in 279 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:47:37.746116 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=30 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:47:38.028679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:53:48.508566 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531b8dbd-0818-4848-9ebf-1f73e950fa0c None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:53:48.509197 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531b8dbd-0818-4848-9ebf-1f73e950fa0c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:53:48.509386 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531b8dbd-0818-4848-9ebf-1f73e950fa0c None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:53:48.511156 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 650/1307] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:53:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:53:48.518730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38501c8-607b-4a52-88f4-02532b38e82f None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:53:48.519455 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38501c8-607b-4a52-88f4-02532b38e82f None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:53:48.519455 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38501c8-607b-4a52-88f4-02532b38e82f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:53:48.599198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-e38501c8-607b-4a52-88f4-02532b38e82f None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:53:48.600665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-e38501c8-607b-4a52-88f4-02532b38e82f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:53:48.714216 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 658/1308] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:53:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:53:49.007734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc7e3ec-9dcd-466e-a04d-525eee971743 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:53:49.008070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc7e3ec-9dcd-466e-a04d-525eee971743 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:53:49.008290 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc7e3ec-9dcd-466e-a04d-525eee971743 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:53:49.009785 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 651/1309] 10.222.0.62 () {54 vars in 964 bytes} [Fri Sep 23 13:53:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:53:49.016434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a23369-265b-49a7-8d43-6c72b2ee1ff7 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:53:49.016791 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a23369-265b-49a7-8d43-6c72b2ee1ff7 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:53:49.017143 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a23369-265b-49a7-8d43-6c72b2ee1ff7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:53:49.099464 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-64a23369-265b-49a7-8d43-6c72b2ee1ff7 None None] MFA Rules not processed for user `580a35627c234a25aa04f5d376177ca5`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:53:49.100790 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-64a23369-265b-49a7-8d43-6c72b2ee1ff7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:53:49.123734 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:53:49.181924 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 659/1310] 10.222.0.62 () {58 vars in 1061 bytes} [Fri Sep 23 13:53:49 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:53:49.191447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc7e3ec-9dcd-466e-a04d-525eee971743 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:53:49.193122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-0cc7e3ec-9dcd-466e-a04d-525eee971743 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:53:49.317413 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc7e3ec-9dcd-466e-a04d-525eee971743 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:53:49.399664 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ab52f1-ca7f-476e-ae91-b09c8c3ae5f6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:53:49.400311 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ab52f1-ca7f-476e-ae91-b09c8c3ae5f6 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:53:49.400591 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ab52f1-ca7f-476e-ae91-b09c8c3ae5f6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:53:49.400862 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ab52f1-ca7f-476e-ae91-b09c8c3ae5f6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:53:49.402499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-43ab52f1-ca7f-476e-ae91-b09c8c3ae5f6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:53:49.463840 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:53:49.521868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ab52f1-ca7f-476e-ae91-b09c8c3ae5f6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:53:49.523427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ab52f1-ca7f-476e-ae91-b09c8c3ae5f6 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:53:49.543783 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 652/1311] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:53:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:53:52.030632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c72e3f0-492f-4a2e-babe-4b08cc965502 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:53:52.031191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c72e3f0-492f-4a2e-babe-4b08cc965502 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:53:52.031191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c72e3f0-492f-4a2e-babe-4b08cc965502 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:53:52.032590 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 660/1312] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:53:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:53:52.037827 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e13de7-fbe1-48dd-a770-cdf79090f34f None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:53:52.038235 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e13de7-fbe1-48dd-a770-cdf79090f34f None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:53:52.038400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e13de7-fbe1-48dd-a770-cdf79090f34f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:53:52.119419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-d8e13de7-fbe1-48dd-a770-cdf79090f34f None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:53:52.120638 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d8e13de7-fbe1-48dd-a770-cdf79090f34f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:53:52.218744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 653/1313] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:53:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:53:52.510839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c72e3f0-492f-4a2e-babe-4b08cc965502 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:53:52.512624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-7c72e3f0-492f-4a2e-babe-4b08cc965502 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:53:52.635292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c72e3f0-492f-4a2e-babe-4b08cc965502 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:53:52.717158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:53:52.719945 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31ab756-8416-4b28-aed5-d71dfdb71514 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:53:52.720548 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31ab756-8416-4b28-aed5-d71dfdb71514 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:53:52.720907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31ab756-8416-4b28-aed5-d71dfdb71514 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:53:52.721296 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31ab756-8416-4b28-aed5-d71dfdb71514 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:53:52.723083 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c31ab756-8416-4b28-aed5-d71dfdb71514 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:53:52.843782 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c31ab756-8416-4b28-aed5-d71dfdb71514 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:53:52.845414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c31ab756-8416-4b28-aed5-d71dfdb71514 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:53:52.865242 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 661/1314] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:53:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:53:55.615178 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a8278d-b563-4c78-a347-3bfe6a2973c0 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:53:55.615670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a8278d-b563-4c78-a347-3bfe6a2973c0 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:53:55.615781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a8278d-b563-4c78-a347-3bfe6a2973c0 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:53:55.617074 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 654/1315] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:53:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:53:55.623125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92044435-06ad-4b25-85c4-a81af76a5372 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:53:55.623543 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92044435-06ad-4b25-85c4-a81af76a5372 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:53:55.623679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92044435-06ad-4b25-85c4-a81af76a5372 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:53:55.690520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-92044435-06ad-4b25-85c4-a81af76a5372 None None] MFA Rules not processed for user `15dbb8e9805348eb9d1237a19e8e15f2`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:53:55.691719 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-92044435-06ad-4b25-85c4-a81af76a5372 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:53:55.743939 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:53:55.787239 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 662/1316] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:53:55 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:53:56.071059 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a8278d-b563-4c78-a347-3bfe6a2973c0 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:53:56.072607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-30a8278d-b563-4c78-a347-3bfe6a2973c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:53:56.082537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:53:56.175234 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a8278d-b563-4c78-a347-3bfe6a2973c0 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:53:56.248283 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5028b04-e69c-47d3-a2fc-229843ecbf7a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:53:56.248952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5028b04-e69c-47d3-a2fc-229843ecbf7a service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:53:56.249235 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5028b04-e69c-47d3-a2fc-229843ecbf7a service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:53:56.249508 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5028b04-e69c-47d3-a2fc-229843ecbf7a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:53:56.251118 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e5028b04-e69c-47d3-a2fc-229843ecbf7a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:53:56.313091 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:53:56.359005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5028b04-e69c-47d3-a2fc-229843ecbf7a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:53:56.360373 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5028b04-e69c-47d3-a2fc-229843ecbf7a service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:53:56.377984 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 655/1317] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:53:56 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:53:59.430851 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62941783-d204-484b-a2f3-0508909eac82 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:53:59.431438 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62941783-d204-484b-a2f3-0508909eac82 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:53:59.431438 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62941783-d204-484b-a2f3-0508909eac82 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:53:59.432737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 663/1318] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:53:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:53:59.438052 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761584cb-102c-433c-9f39-1800d3f16cd3 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:53:59.438469 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761584cb-102c-433c-9f39-1800d3f16cd3 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:53:59.438635 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761584cb-102c-433c-9f39-1800d3f16cd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:53:59.503727 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-761584cb-102c-433c-9f39-1800d3f16cd3 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:53:59.504870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-761584cb-102c-433c-9f39-1800d3f16cd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:53:59.599002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 656/1319] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:53:59 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:53:59.887045 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62941783-d204-484b-a2f3-0508909eac82 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:53:59.889116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-62941783-d204-484b-a2f3-0508909eac82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:53:59.993127 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62941783-d204-484b-a2f3-0508909eac82 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:54:00.067187 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999d128d-9ef4-4ca0-8058-9c399afdf9f6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:54:00.067783 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999d128d-9ef4-4ca0-8058-9c399afdf9f6 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:00.068055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999d128d-9ef4-4ca0-8058-9c399afdf9f6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:00.068317 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999d128d-9ef4-4ca0-8058-9c399afdf9f6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:00.069882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-999d128d-9ef4-4ca0-8058-9c399afdf9f6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:00.077155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:00.175542 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999d128d-9ef4-4ca0-8058-9c399afdf9f6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:54:00.176880 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999d128d-9ef4-4ca0-8058-9c399afdf9f6 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:54:00.194830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 664/1320] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:53:59 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:54:03.343959 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ef033e-6606-4ddf-a519-822e326b3ad9 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:03.344509 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ef033e-6606-4ddf-a519-822e326b3ad9 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:03.344509 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ef033e-6606-4ddf-a519-822e326b3ad9 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:03.345903 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 657/1321] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:54:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:54:03.351351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43127ef-9852-440c-b5bf-a10037bb5bcc None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:03.351920 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43127ef-9852-440c-b5bf-a10037bb5bcc None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:03.352129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43127ef-9852-440c-b5bf-a10037bb5bcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:03.419425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-a43127ef-9852-440c-b5bf-a10037bb5bcc None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:54:03.420653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a43127ef-9852-440c-b5bf-a10037bb5bcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:03.481833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:03.520712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 665/1322] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:54:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:54:03.821095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee37d196-de26-4129-a254-2099bcb3ca25 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:03.821492 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee37d196-de26-4129-a254-2099bcb3ca25 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:03.821868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee37d196-de26-4129-a254-2099bcb3ca25 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:03.823340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 658/1323] 10.222.0.62 () {54 vars in 964 bytes} [Fri Sep 23 13:54:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:54:03.829888 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3bdc42-c3c2-4056-a3c0-f6d51382e8de None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:03.830244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3bdc42-c3c2-4056-a3c0-f6d51382e8de None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:03.830618 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3bdc42-c3c2-4056-a3c0-f6d51382e8de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:03.909209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-cd3bdc42-c3c2-4056-a3c0-f6d51382e8de None None] MFA Rules not processed for user `580a35627c234a25aa04f5d376177ca5`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:54:03.910453 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-cd3bdc42-c3c2-4056-a3c0-f6d51382e8de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:03.996452 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 666/1324] 10.222.0.62 () {58 vars in 1061 bytes} [Fri Sep 23 13:54:03 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 13:54:04.005601 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee37d196-de26-4129-a254-2099bcb3ca25 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:54:04.007155 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ee37d196-de26-4129-a254-2099bcb3ca25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:04.027214 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:04.113694 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee37d196-de26-4129-a254-2099bcb3ca25 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:54:04.189339 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5d9901-02f7-4485-a9db-c97f8dd07993 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:54:04.189939 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5d9901-02f7-4485-a9db-c97f8dd07993 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:04.190213 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5d9901-02f7-4485-a9db-c97f8dd07993 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:04.190515 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5d9901-02f7-4485-a9db-c97f8dd07993 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:04.192093 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-bd5d9901-02f7-4485-a9db-c97f8dd07993 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:04.268368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:04.306137 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5d9901-02f7-4485-a9db-c97f8dd07993 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:54:04.307580 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd5d9901-02f7-4485-a9db-c97f8dd07993 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:54:04.326064 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 659/1325] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:54:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:54:06.761695 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d1f1b1-5364-401e-a73b-042d679d273b None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:06.762222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d1f1b1-5364-401e-a73b-042d679d273b None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:06.762385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d1f1b1-5364-401e-a73b-042d679d273b None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:06.763646 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 667/1326] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:54:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:54:06.768851 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a2ff3c-5871-486b-b0dc-3c12f2924093 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:06.769295 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a2ff3c-5871-486b-b0dc-3c12f2924093 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:06.769397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a2ff3c-5871-486b-b0dc-3c12f2924093 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:06.836966 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-74a2ff3c-5871-486b-b0dc-3c12f2924093 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:54:06.838175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-74a2ff3c-5871-486b-b0dc-3c12f2924093 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:06.934254 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 660/1327] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:54:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:54:07.217282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d1f1b1-5364-401e-a73b-042d679d273b None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:54:07.219108 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-16d1f1b1-5364-401e-a73b-042d679d273b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:07.279424 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:07.322949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d1f1b1-5364-401e-a73b-042d679d273b None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:54:07.395354 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ebcd84-33e2-4a30-8efe-afda44fdd319 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:54:07.395953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ebcd84-33e2-4a30-8efe-afda44fdd319 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:07.396225 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ebcd84-33e2-4a30-8efe-afda44fdd319 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:07.396496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ebcd84-33e2-4a30-8efe-afda44fdd319 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:07.398065 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-54ebcd84-33e2-4a30-8efe-afda44fdd319 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:07.503926 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:07.506388 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ebcd84-33e2-4a30-8efe-afda44fdd319 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:54:07.507819 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ebcd84-33e2-4a30-8efe-afda44fdd319 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:54:07.525995 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 668/1328] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:54:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:54:09.753362 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5c3cb0-fb99-474f-8712-90d198048d87 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:09.753861 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5c3cb0-fb99-474f-8712-90d198048d87 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:09.753964 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5c3cb0-fb99-474f-8712-90d198048d87 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:09.755373 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 661/1329] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:54:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:54:09.760988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756d2530-6bb1-4ff2-a9a9-6b73bf3aaf82 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:09.761659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756d2530-6bb1-4ff2-a9a9-6b73bf3aaf82 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:09.761659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756d2530-6bb1-4ff2-a9a9-6b73bf3aaf82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:09.827385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-756d2530-6bb1-4ff2-a9a9-6b73bf3aaf82 None None] MFA Rules not processed for user `15dbb8e9805348eb9d1237a19e8e15f2`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:54:09.828621 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-756d2530-6bb1-4ff2-a9a9-6b73bf3aaf82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:09.937068 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 669/1330] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:54:09 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:54:10.221302 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5c3cb0-fb99-474f-8712-90d198048d87 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:54:10.222930 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-0e5c3cb0-fb99-474f-8712-90d198048d87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:10.253622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:10.326929 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5c3cb0-fb99-474f-8712-90d198048d87 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:54:10.401123 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e16db7d-02f8-4fc6-9e8f-c9d7ac924706 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:54:10.402287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e16db7d-02f8-4fc6-9e8f-c9d7ac924706 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:10.402640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e16db7d-02f8-4fc6-9e8f-c9d7ac924706 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:10.402911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e16db7d-02f8-4fc6-9e8f-c9d7ac924706 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:10.404512 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-7e16db7d-02f8-4fc6-9e8f-c9d7ac924706 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:10.483496 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:10.514512 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e16db7d-02f8-4fc6-9e8f-c9d7ac924706 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:54:10.515947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e16db7d-02f8-4fc6-9e8f-c9d7ac924706 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:54:10.535018 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 662/1331] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:54:10 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:54:13.330096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516eb831-6bd6-4b92-a895-628c7e474329 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:13.330712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516eb831-6bd6-4b92-a895-628c7e474329 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:13.330712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516eb831-6bd6-4b92-a895-628c7e474329 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:13.332020 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 670/1332] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:54:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:54:13.337176 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1aa5b5-6f3d-4ef5-8953-9729d858e588 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:13.337598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1aa5b5-6f3d-4ef5-8953-9729d858e588 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:13.337711 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1aa5b5-6f3d-4ef5-8953-9729d858e588 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:13.406622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-1d1aa5b5-6f3d-4ef5-8953-9729d858e588 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:54:13.407837 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-1d1aa5b5-6f3d-4ef5-8953-9729d858e588 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:13.502740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 663/1333] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:54:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:54:13.785071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516eb831-6bd6-4b92-a895-628c7e474329 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:54:13.786637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-516eb831-6bd6-4b92-a895-628c7e474329 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:13.856501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:13.891156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516eb831-6bd6-4b92-a895-628c7e474329 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:54:13.963609 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aafca4d-e41a-4bb4-af2b-f2912748bc39 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:54:13.964283 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aafca4d-e41a-4bb4-af2b-f2912748bc39 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:13.964562 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aafca4d-e41a-4bb4-af2b-f2912748bc39 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:13.964833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aafca4d-e41a-4bb4-af2b-f2912748bc39 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:13.966433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-5aafca4d-e41a-4bb4-af2b-f2912748bc39 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:14.072595 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aafca4d-e41a-4bb4-af2b-f2912748bc39 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:54:14.074101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aafca4d-e41a-4bb4-af2b-f2912748bc39 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:54:14.083770 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:14.091653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 671/1334] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:54:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:54:16.271875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610c4cd2-3d2c-438b-86d4-358d9a86dd78 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:16.272393 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610c4cd2-3d2c-438b-86d4-358d9a86dd78 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:16.272487 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610c4cd2-3d2c-438b-86d4-358d9a86dd78 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:16.273774 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 664/1335] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:54:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:54:16.279564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bc6a43-5dc9-4219-baa3-398b206521cb None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:16.280069 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bc6a43-5dc9-4219-baa3-398b206521cb None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:16.280175 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bc6a43-5dc9-4219-baa3-398b206521cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:16.346803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-49bc6a43-5dc9-4219-baa3-398b206521cb None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:54:16.348004 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-49bc6a43-5dc9-4219-baa3-398b206521cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:16.443229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 672/1336] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:54:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:54:16.730120 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610c4cd2-3d2c-438b-86d4-358d9a86dd78 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:54:16.731690 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-610c4cd2-3d2c-438b-86d4-358d9a86dd78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:16.772290 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:16.833525 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610c4cd2-3d2c-438b-86d4-358d9a86dd78 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:54:16.905279 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a9367b-2aa4-4545-a319-1df19aa1884c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:54:16.905870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a9367b-2aa4-4545-a319-1df19aa1884c service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:16.906143 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a9367b-2aa4-4545-a319-1df19aa1884c service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:16.906431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a9367b-2aa4-4545-a319-1df19aa1884c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:16.907989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e2a9367b-2aa4-4545-a319-1df19aa1884c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:16.989297 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:17.014199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a9367b-2aa4-4545-a319-1df19aa1884c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:54:17.015631 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a9367b-2aa4-4545-a319-1df19aa1884c service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:54:17.033603 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 665/1337] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:54:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:54:20.002034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5e6d66-5f38-47d9-b309-3c1746db5926 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:20.002585 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5e6d66-5f38-47d9-b309-3c1746db5926 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:20.002678 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5e6d66-5f38-47d9-b309-3c1746db5926 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:20.003965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 673/1338] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:54:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:54:20.009185 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5dc90f6-3f9b-4268-8595-d110a0b9ea49 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:20.009607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5dc90f6-3f9b-4268-8595-d110a0b9ea49 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:20.009745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5dc90f6-3f9b-4268-8595-d110a0b9ea49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:20.076581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-d5dc90f6-3f9b-4268-8595-d110a0b9ea49 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:54:20.077912 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d5dc90f6-3f9b-4268-8595-d110a0b9ea49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:20.175112 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 666/1339] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:54:20 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:54:20.459170 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5e6d66-5f38-47d9-b309-3c1746db5926 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:54:20.460685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-2a5e6d66-5f38-47d9-b309-3c1746db5926 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:20.535418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:20.563962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5e6d66-5f38-47d9-b309-3c1746db5926 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:54:20.638494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2252ab-6cbb-4f24-b94f-c203e0276e13 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:54:20.639098 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2252ab-6cbb-4f24-b94f-c203e0276e13 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:20.639371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2252ab-6cbb-4f24-b94f-c203e0276e13 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:20.639763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2252ab-6cbb-4f24-b94f-c203e0276e13 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:20.642189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-cf2252ab-6cbb-4f24-b94f-c203e0276e13 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:20.753593 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2252ab-6cbb-4f24-b94f-c203e0276e13 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:54:20.755012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2252ab-6cbb-4f24-b94f-c203e0276e13 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:54:20.772944 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 674/1340] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:54:20 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:54:25.194800 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1414839c-8780-46f0-8235-dbfdf133bfe2 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:25.195328 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1414839c-8780-46f0-8235-dbfdf133bfe2 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:25.195328 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1414839c-8780-46f0-8235-dbfdf133bfe2 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:25.196836 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 667/1341] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:54:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:54:25.202818 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8458df3-5bf3-4f85-b3f3-bb77667561e8 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:25.203285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8458df3-5bf3-4f85-b3f3-bb77667561e8 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:25.203386 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8458df3-5bf3-4f85-b3f3-bb77667561e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:25.215482 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:25.280767 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-e8458df3-5bf3-4f85-b3f3-bb77667561e8 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:54:25.281960 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-e8458df3-5bf3-4f85-b3f3-bb77667561e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:25.380515 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 675/1342] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:54:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:54:25.667617 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1414839c-8780-46f0-8235-dbfdf133bfe2 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:54:25.669228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-1414839c-8780-46f0-8235-dbfdf133bfe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:25.718433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:25.774016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1414839c-8780-46f0-8235-dbfdf133bfe2 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:54:25.849023 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ec1bf1-7f7d-422d-a740-9f7ec6a72a1c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:54:25.849622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ec1bf1-7f7d-422d-a740-9f7ec6a72a1c service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:25.849898 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ec1bf1-7f7d-422d-a740-9f7ec6a72a1c service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:25.850160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ec1bf1-7f7d-422d-a740-9f7ec6a72a1c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:25.851764 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-49ec1bf1-7f7d-422d-a740-9f7ec6a72a1c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:25.949947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:25.967491 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ec1bf1-7f7d-422d-a740-9f7ec6a72a1c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:54:25.968904 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ec1bf1-7f7d-422d-a740-9f7ec6a72a1c service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:54:25.987425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 668/1343] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:54:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:54:29.152170 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9135b764-8754-42c5-9145-fb22928b89c4 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:29.152813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9135b764-8754-42c5-9145-fb22928b89c4 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:29.152813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9135b764-8754-42c5-9145-fb22928b89c4 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:29.154067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 676/1344] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:54:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:54:29.159352 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec619fa5-596e-488a-a314-fd6bcf450e64 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:29.159798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec619fa5-596e-488a-a314-fd6bcf450e64 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:29.159899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec619fa5-596e-488a-a314-fd6bcf450e64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:29.228239 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-ec619fa5-596e-488a-a314-fd6bcf450e64 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:54:29.229475 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ec619fa5-596e-488a-a314-fd6bcf450e64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:29.339693 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 669/1345] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:54:29 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:54:29.629200 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9135b764-8754-42c5-9145-fb22928b89c4 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:54:29.630834 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-9135b764-8754-42c5-9145-fb22928b89c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:29.720303 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:29.736934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9135b764-8754-42c5-9145-fb22928b89c4 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:54:29.814104 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81575536-9d33-40e6-9735-3adc2056b36e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:54:29.815388 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81575536-9d33-40e6-9735-3adc2056b36e service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:29.815388 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81575536-9d33-40e6-9735-3adc2056b36e service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:29.815388 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81575536-9d33-40e6-9735-3adc2056b36e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:29.816858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-81575536-9d33-40e6-9735-3adc2056b36e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:29.950662 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81575536-9d33-40e6-9735-3adc2056b36e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:54:29.952112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81575536-9d33-40e6-9735-3adc2056b36e service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:54:29.974170 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 677/1346] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:54:29 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:54:34.373250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880ce84e-2c4c-40bd-b717-df5c36785328 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:34.373767 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880ce84e-2c4c-40bd-b717-df5c36785328 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:34.373767 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880ce84e-2c4c-40bd-b717-df5c36785328 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:34.375178 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 670/1347] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:54:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:54:34.380336 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51421340-095f-4d25-a865-01a06cbc97ff None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:34.380739 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51421340-095f-4d25-a865-01a06cbc97ff None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:34.380890 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51421340-095f-4d25-a865-01a06cbc97ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:34.396358 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:34.448658 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-51421340-095f-4d25-a865-01a06cbc97ff None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:54:34.449841 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-51421340-095f-4d25-a865-01a06cbc97ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:34.551341 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 678/1348] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:54:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:54:34.833528 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880ce84e-2c4c-40bd-b717-df5c36785328 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:54:34.835106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-880ce84e-2c4c-40bd-b717-df5c36785328 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:34.894315 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:34.939485 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880ce84e-2c4c-40bd-b717-df5c36785328 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:54:35.013213 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2acb2b3-9a61-4cbc-b53c-29c9a4171a76 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:54:35.013815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2acb2b3-9a61-4cbc-b53c-29c9a4171a76 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:35.014098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2acb2b3-9a61-4cbc-b53c-29c9a4171a76 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:35.014392 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2acb2b3-9a61-4cbc-b53c-29c9a4171a76 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:35.015978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a2acb2b3-9a61-4cbc-b53c-29c9a4171a76 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:35.122275 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:35.124456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2acb2b3-9a61-4cbc-b53c-29c9a4171a76 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:54:35.125875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2acb2b3-9a61-4cbc-b53c-29c9a4171a76 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:54:35.144804 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 671/1349] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:54:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:54:40.183597 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d789a16-21d1-482e-bea8-b326f01c8d94 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:40.184002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d789a16-21d1-482e-bea8-b326f01c8d94 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:40.184206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d789a16-21d1-482e-bea8-b326f01c8d94 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:40.185923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 679/1350] 10.222.0.62 () {54 vars in 921 bytes} [Fri Sep 23 13:54:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:54:40.194562 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f27ac2-7672-46c8-af97-b556ce2416ff None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:40.195005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f27ac2-7672-46c8-af97-b556ce2416ff None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:40.195089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f27ac2-7672-46c8-af97-b556ce2416ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:40.304039 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-c1f27ac2-7672-46c8-af97-b556ce2416ff None None] MFA Rules not processed for user `ce249d4a99c34891b6ba941f57a14dd4`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:54:40.305578 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-c1f27ac2-7672-46c8-af97-b556ce2416ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:40.489941 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 672/1351] 10.222.0.62 () {58 vars in 1018 bytes} [Fri Sep 23 13:54:40 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 300 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:54:44.549993 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=59 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:44.555863 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc4f8b3-d149-4f2a-a0ad-72b13bba734a None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:44.556141 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc4f8b3-d149-4f2a-a0ad-72b13bba734a None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:44.556411 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc4f8b3-d149-4f2a-a0ad-72b13bba734a None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:44.558268 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 680/1352] 10.222.0.62 () {54 vars in 921 bytes} [Fri Sep 23 13:54:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:54:44.563657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b17127b-fc89-41e6-9d5b-57d3d9408182 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:44.567523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b17127b-fc89-41e6-9d5b-57d3d9408182 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:44.567523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b17127b-fc89-41e6-9d5b-57d3d9408182 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:44.623291 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:44.651547 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-6b17127b-fc89-41e6-9d5b-57d3d9408182 None None] MFA Rules not processed for user `ce249d4a99c34891b6ba941f57a14dd4`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:54:44.652771 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-6b17127b-fc89-41e6-9d5b-57d3d9408182 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:44.769920 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 673/1353] 10.222.0.62 () {58 vars in 1018 bytes} [Fri Sep 23 13:54:44 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:54:46.327209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7739d57-c076-4fcc-b5f1-40dab18875c9 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:46.327750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7739d57-c076-4fcc-b5f1-40dab18875c9 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:46.327750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7739d57-c076-4fcc-b5f1-40dab18875c9 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:46.329111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 681/1354] 10.222.0.62 () {54 vars in 921 bytes} [Fri Sep 23 13:54:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:54:46.334047 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5df4423-c8c8-45cc-99bb-8acf00660a2e None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:46.334493 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5df4423-c8c8-45cc-99bb-8acf00660a2e None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:46.334734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5df4423-c8c8-45cc-99bb-8acf00660a2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:46.411884 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-f5df4423-c8c8-45cc-99bb-8acf00660a2e None None] MFA Rules not processed for user `ce249d4a99c34891b6ba941f57a14dd4`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:54:46.413121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f5df4423-c8c8-45cc-99bb-8acf00660a2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:46.456600 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:46.522471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 674/1355] 10.222.0.62 () {58 vars in 1018 bytes} [Fri Sep 23 13:54:46 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 13:54:50.078405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ce6601-2df9-4c2d-a1de-470f7bed0f76 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:50.079730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ce6601-2df9-4c2d-a1de-470f7bed0f76 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:50.080246 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ce6601-2df9-4c2d-a1de-470f7bed0f76 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:50.082060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 682/1356] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:54:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:54:50.087934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99ed754-402d-4194-8750-377905626ffc None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:50.088863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99ed754-402d-4194-8750-377905626ffc None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:50.089306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99ed754-402d-4194-8750-377905626ffc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:50.181234 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-d99ed754-402d-4194-8750-377905626ffc None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:54:50.183070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d99ed754-402d-4194-8750-377905626ffc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:50.289628 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 675/1357] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:54:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:54:50.596512 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbf9494-56db-498e-868f-074d980aa0fe None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:50.597005 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbf9494-56db-498e-868f-074d980aa0fe None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:50.597431 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbf9494-56db-498e-868f-074d980aa0fe None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:50.600084 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 683/1358] 10.222.0.62 () {54 vars in 961 bytes} [Fri Sep 23 13:54:50 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:54:50.609246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531028d2-17c7-43bd-9054-36dcdd402d39 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:50.610145 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531028d2-17c7-43bd-9054-36dcdd402d39 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:50.610747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531028d2-17c7-43bd-9054-36dcdd402d39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:50.627927 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:50.696435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-531028d2-17c7-43bd-9054-36dcdd402d39 None None] MFA Rules not processed for user `f01ed94dbb384a178ad0a45a6f3d8c17`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:54:50.698468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-531028d2-17c7-43bd-9054-36dcdd402d39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:50.819968 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 676/1359] 10.222.0.62 () {58 vars in 1058 bytes} [Fri Sep 23 13:54:50 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:54:50.828461 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbf9494-56db-498e-868f-074d980aa0fe None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:54:50.830087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-fcbf9494-56db-498e-868f-074d980aa0fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:50.988907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbf9494-56db-498e-868f-074d980aa0fe None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:54:51.074719 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:51.113391 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beda2601-b221-43d2-9c98-c415ca770eeb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:54:51.114188 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beda2601-b221-43d2-9c98-c415ca770eeb service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:51.114585 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beda2601-b221-43d2-9c98-c415ca770eeb service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:51.114953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beda2601-b221-43d2-9c98-c415ca770eeb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:51.116655 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-beda2601-b221-43d2-9c98-c415ca770eeb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:51.217334 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beda2601-b221-43d2-9c98-c415ca770eeb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:54:51.219237 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beda2601-b221-43d2-9c98-c415ca770eeb service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:54:51.238726 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 684/1360] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:54:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:54:53.623364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55d06b2-445d-44ea-bb74-b1de82c61c4d None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:53.623364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55d06b2-445d-44ea-bb74-b1de82c61c4d None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:53.623364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55d06b2-445d-44ea-bb74-b1de82c61c4d None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:53.623364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 677/1361] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:54:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:54:53.632738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d692a215-3225-422c-8d19-8b95c09d0785 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:53.632738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d692a215-3225-422c-8d19-8b95c09d0785 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:53.632738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d692a215-3225-422c-8d19-8b95c09d0785 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:53.703917 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-d692a215-3225-422c-8d19-8b95c09d0785 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:54:53.705139 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d692a215-3225-422c-8d19-8b95c09d0785 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:53.749818 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:53.838653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 685/1362] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:54:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:54:54.189311 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55d06b2-445d-44ea-bb74-b1de82c61c4d None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:54:54.190850 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d55d06b2-445d-44ea-bb74-b1de82c61c4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:54.270277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:54.362300 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55d06b2-445d-44ea-bb74-b1de82c61c4d None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:54:54.511795 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1cd6e9-5547-4f7e-996d-5f0249e259f6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:54:54.512438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1cd6e9-5547-4f7e-996d-5f0249e259f6 service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:54.512714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1cd6e9-5547-4f7e-996d-5f0249e259f6 service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:54.512954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1cd6e9-5547-4f7e-996d-5f0249e259f6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:54.514671 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-1e1cd6e9-5547-4f7e-996d-5f0249e259f6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:54.541222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:54.615181 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1cd6e9-5547-4f7e-996d-5f0249e259f6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:54:54.616798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1cd6e9-5547-4f7e-996d-5f0249e259f6 service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:54:54.638804 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 678/1363] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:54:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:54:57.237365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee37d03d-7d85-48e3-ae61-1dcf26845d0b None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:57.237874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee37d03d-7d85-48e3-ae61-1dcf26845d0b None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:57.237959 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee37d03d-7d85-48e3-ae61-1dcf26845d0b None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:57.239433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 686/1364] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:54:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:54:57.244448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081642e9-fefa-418b-9b80-527ea6701bfe None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:57.244839 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081642e9-fefa-418b-9b80-527ea6701bfe None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:57.245020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081642e9-fefa-418b-9b80-527ea6701bfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:57.314425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-081642e9-fefa-418b-9b80-527ea6701bfe None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:54:57.315644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-081642e9-fefa-418b-9b80-527ea6701bfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:57.417738 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:57.427603 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 679/1365] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:54:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:54:57.721564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f94e830-8c99-4899-9549-c98bc3a9cecf None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:57.721856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f94e830-8c99-4899-9549-c98bc3a9cecf None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:57.722156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f94e830-8c99-4899-9549-c98bc3a9cecf None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:57.723689 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 687/1366] 10.222.0.62 () {54 vars in 961 bytes} [Fri Sep 23 13:54:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:54:57.729669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eb2dc8-2884-4932-9363-54d6df2bca4b None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:57.730103 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eb2dc8-2884-4932-9363-54d6df2bca4b None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:57.730470 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eb2dc8-2884-4932-9363-54d6df2bca4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:57.810861 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-97eb2dc8-2884-4932-9363-54d6df2bca4b None None] MFA Rules not processed for user `f01ed94dbb384a178ad0a45a6f3d8c17`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:54:57.812063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-97eb2dc8-2884-4932-9363-54d6df2bca4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:57.938241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 680/1367] 10.222.0.62 () {58 vars in 1058 bytes} [Fri Sep 23 13:54:57 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:54:57.946972 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f94e830-8c99-4899-9549-c98bc3a9cecf None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:54:57.948735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-3f94e830-8c99-4899-9549-c98bc3a9cecf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:58.108135 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f94e830-8c99-4899-9549-c98bc3a9cecf None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:54:58.129768 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:58.236799 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12206e4-ab7e-499c-9a7b-30d0d2b98471 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:54:58.237418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12206e4-ab7e-499c-9a7b-30d0d2b98471 service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:58.237951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12206e4-ab7e-499c-9a7b-30d0d2b98471 service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:58.238043 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12206e4-ab7e-499c-9a7b-30d0d2b98471 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:58.239623 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a12206e4-ab7e-499c-9a7b-30d0d2b98471 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:58.335677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12206e4-ab7e-499c-9a7b-30d0d2b98471 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:54:58.338094 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12206e4-ab7e-499c-9a7b-30d0d2b98471 service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:54:58.361448 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 688/1368] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:54:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:54:58.460763 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607a0883-9fec-4dea-86fa-1aafb64d4c9c None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:58.461317 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607a0883-9fec-4dea-86fa-1aafb64d4c9c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:58.461317 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607a0883-9fec-4dea-86fa-1aafb64d4c9c None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:58.463518 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 681/1369] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:54:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:54:58.469306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f3c556-7d42-4b23-b2e7-c7b9d8130003 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:58.469531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f3c556-7d42-4b23-b2e7-c7b9d8130003 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:58.469802 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f3c556-7d42-4b23-b2e7-c7b9d8130003 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:58.509897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:58.548179 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-40f3c556-7d42-4b23-b2e7-c7b9d8130003 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:54:58.549675 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-40f3c556-7d42-4b23-b2e7-c7b9d8130003 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:58.666962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 689/1370] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:54:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:54:59.004194 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d5afb5-0db7-4d75-b3bc-98dd86dcb066 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:59.004194 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d5afb5-0db7-4d75-b3bc-98dd86dcb066 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:59.004194 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d5afb5-0db7-4d75-b3bc-98dd86dcb066 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:59.006714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 682/1371] 10.222.0.62 () {54 vars in 963 bytes} [Fri Sep 23 13:54:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:54:59.011487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ba0e92-2b62-4105-9ce3-aafa0575afd9 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:59.011777 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ba0e92-2b62-4105-9ce3-aafa0575afd9 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:59.012045 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ba0e92-2b62-4105-9ce3-aafa0575afd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:59.096448 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-02ba0e92-2b62-4105-9ce3-aafa0575afd9 None None] MFA Rules not processed for user `fae68ca0754a4b0b88a57f39efa2fb45`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:54:59.097637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-02ba0e92-2b62-4105-9ce3-aafa0575afd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:59.152209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:59.182600 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 690/1372] 10.222.0.62 () {58 vars in 1060 bytes} [Fri Sep 23 13:54:59 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:54:59.190399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d5afb5-0db7-4d75-b3bc-98dd86dcb066 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:54:59.191888 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-c4d5afb5-0db7-4d75-b3bc-98dd86dcb066 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:59.248671 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:59.303245 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d5afb5-0db7-4d75-b3bc-98dd86dcb066 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:54:59.385684 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7aafe2-dd97-425c-a7ef-451b6df8f42a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:54:59.386294 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7aafe2-dd97-425c-a7ef-451b6df8f42a service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:54:59.386597 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7aafe2-dd97-425c-a7ef-451b6df8f42a service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:54:59.386865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7aafe2-dd97-425c-a7ef-451b6df8f42a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:54:59.388460 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2b7aafe2-dd97-425c-a7ef-451b6df8f42a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:54:59.492677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:54:59.507097 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7aafe2-dd97-425c-a7ef-451b6df8f42a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:54:59.508483 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7aafe2-dd97-425c-a7ef-451b6df8f42a service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:54:59.527376 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 683/1373] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:54:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 13:55:00.770087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e689c6b9-b93c-48ce-9bd6-503ea4ab4ffb None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:00.770729 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e689c6b9-b93c-48ce-9bd6-503ea4ab4ffb None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:00.770729 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e689c6b9-b93c-48ce-9bd6-503ea4ab4ffb None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:00.772011 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 691/1374] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:55:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:55:00.777777 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4ea26a-cdc1-497b-98bb-4f7976e2484c None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:00.778186 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4ea26a-cdc1-497b-98bb-4f7976e2484c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:00.778352 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4ea26a-cdc1-497b-98bb-4f7976e2484c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:00.844444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-2f4ea26a-cdc1-497b-98bb-4f7976e2484c None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:00.845653 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2f4ea26a-cdc1-497b-98bb-4f7976e2484c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:00.943360 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 684/1375] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:55:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:01.239073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e689c6b9-b93c-48ce-9bd6-503ea4ab4ffb None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:55:01.240651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-e689c6b9-b93c-48ce-9bd6-503ea4ab4ffb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:01.402225 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e689c6b9-b93c-48ce-9bd6-503ea4ab4ffb None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:55:01.461012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:01.525704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf6d6f1-e01f-41b2-aa47-44879afe00d3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:55:01.526311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf6d6f1-e01f-41b2-aa47-44879afe00d3 service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:01.526622 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf6d6f1-e01f-41b2-aa47-44879afe00d3 service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:01.526889 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf6d6f1-e01f-41b2-aa47-44879afe00d3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:01.528486 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-6bf6d6f1-e01f-41b2-aa47-44879afe00d3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:01.622965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf6d6f1-e01f-41b2-aa47-44879afe00d3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:55:01.624594 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf6d6f1-e01f-41b2-aa47-44879afe00d3 service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:55:01.643535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 692/1376] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:55:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:55:04.139423 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d4b34c-c9c7-41a3-9cfe-8354130f8dce None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:04.139995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d4b34c-c9c7-41a3-9cfe-8354130f8dce None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:04.139995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d4b34c-c9c7-41a3-9cfe-8354130f8dce None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:04.141302 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 685/1377] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:55:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:55:04.146601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57756e0a-e85a-4a93-beba-b80ef0f4f4d1 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:04.146995 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57756e0a-e85a-4a93-beba-b80ef0f4f4d1 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:04.147193 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57756e0a-e85a-4a93-beba-b80ef0f4f4d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:04.222667 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-57756e0a-e85a-4a93-beba-b80ef0f4f4d1 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:04.223924 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-57756e0a-e85a-4a93-beba-b80ef0f4f4d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:04.231394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:04.332860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 693/1378] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:55:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:04.621346 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d4b34c-c9c7-41a3-9cfe-8354130f8dce None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:55:04.623146 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-57d4b34c-c9c7-41a3-9cfe-8354130f8dce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:04.691111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:04.784186 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d4b34c-c9c7-41a3-9cfe-8354130f8dce None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:55:04.908694 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a681355-e0c4-4d52-b233-f2d714d6bc3c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:55:04.909415 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a681355-e0c4-4d52-b233-f2d714d6bc3c service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:04.909804 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a681355-e0c4-4d52-b233-f2d714d6bc3c service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:04.910194 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a681355-e0c4-4d52-b233-f2d714d6bc3c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:04.911896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9a681355-e0c4-4d52-b233-f2d714d6bc3c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:04.934862 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:05.011349 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a681355-e0c4-4d52-b233-f2d714d6bc3c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:55:05.013032 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a681355-e0c4-4d52-b233-f2d714d6bc3c service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:55:05.033730 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 686/1379] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:55:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:55:07.427244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1af2497-1c91-4c8f-b867-213321ffc1c7 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:07.427785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1af2497-1c91-4c8f-b867-213321ffc1c7 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:07.427785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1af2497-1c91-4c8f-b867-213321ffc1c7 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:07.429348 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 694/1380] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:55:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:55:07.435411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c146b9a-2ebd-45cd-8737-e6673bffd307 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:07.435910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c146b9a-2ebd-45cd-8737-e6673bffd307 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:07.436039 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c146b9a-2ebd-45cd-8737-e6673bffd307 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:07.511188 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-5c146b9a-2ebd-45cd-8737-e6673bffd307 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:07.512352 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-5c146b9a-2ebd-45cd-8737-e6673bffd307 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:07.606597 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:07.612300 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 687/1381] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:55:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:07.918037 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1af2497-1c91-4c8f-b867-213321ffc1c7 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:55:07.919708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-e1af2497-1c91-4c8f-b867-213321ffc1c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:08.063248 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:08.077700 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1af2497-1c91-4c8f-b867-213321ffc1c7 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:55:08.208087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71232c16-7158-454d-9ad2-b6e882b34d88 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:55:08.208724 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71232c16-7158-454d-9ad2-b6e882b34d88 service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:08.208995 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71232c16-7158-454d-9ad2-b6e882b34d88 service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:08.209315 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71232c16-7158-454d-9ad2-b6e882b34d88 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:08.211100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-71232c16-7158-454d-9ad2-b6e882b34d88 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:08.330032 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71232c16-7158-454d-9ad2-b6e882b34d88 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:55:08.332521 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71232c16-7158-454d-9ad2-b6e882b34d88 service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:55:08.348019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:08.359920 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 695/1382] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:55:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:55:10.756697 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47981dcc-385f-441c-8aea-bd56f3f440d3 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:10.757275 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47981dcc-385f-441c-8aea-bd56f3f440d3 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:10.757275 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47981dcc-385f-441c-8aea-bd56f3f440d3 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:10.758778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 688/1383] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:55:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:55:10.764800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b13f23-8ab1-41a2-8f3e-1a18e53b109e None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:10.765217 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b13f23-8ab1-41a2-8f3e-1a18e53b109e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:10.765527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b13f23-8ab1-41a2-8f3e-1a18e53b109e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:10.863920 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-d0b13f23-8ab1-41a2-8f3e-1a18e53b109e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:10.865843 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d0b13f23-8ab1-41a2-8f3e-1a18e53b109e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:10.988869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 696/1384] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:55:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:11.328140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47981dcc-385f-441c-8aea-bd56f3f440d3 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:55:11.329693 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-47981dcc-385f-441c-8aea-bd56f3f440d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:11.497520 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47981dcc-385f-441c-8aea-bd56f3f440d3 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:55:11.575672 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:11.648028 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5b43ad5-dba7-4c8b-bfcf-8db75015182b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:55:11.649673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b43ad5-dba7-4c8b-bfcf-8db75015182b service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:11.650082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b43ad5-dba7-4c8b-bfcf-8db75015182b service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:11.650853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b43ad5-dba7-4c8b-bfcf-8db75015182b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:11.653281 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e5b43ad5-dba7-4c8b-bfcf-8db75015182b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:11.750583 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b43ad5-dba7-4c8b-bfcf-8db75015182b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:55:11.752072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5b43ad5-dba7-4c8b-bfcf-8db75015182b service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:55:11.771112 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 689/1385] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:55:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:55:14.172798 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577eebcb-1d22-46ba-bbbf-5761a353743d None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:14.173350 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577eebcb-1d22-46ba-bbbf-5761a353743d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:14.173350 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577eebcb-1d22-46ba-bbbf-5761a353743d None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:14.174658 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 697/1386] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:55:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:55:14.180687 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1e9863-d3ce-407c-81ca-d9744f211430 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:14.181134 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1e9863-d3ce-407c-81ca-d9744f211430 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:14.181232 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d1e9863-d3ce-407c-81ca-d9744f211430 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:14.251919 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-1d1e9863-d3ce-407c-81ca-d9744f211430 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:14.253123 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-1d1e9863-d3ce-407c-81ca-d9744f211430 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:14.288536 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:14.353770 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 690/1387] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:55:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:14.650984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577eebcb-1d22-46ba-bbbf-5761a353743d None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:55:14.652522 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-577eebcb-1d22-46ba-bbbf-5761a353743d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:14.731362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:14.801969 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577eebcb-1d22-46ba-bbbf-5761a353743d None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:55:14.921192 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8d09d2-48f7-4f79-a0a0-635605afb414 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:55:14.921710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8d09d2-48f7-4f79-a0a0-635605afb414 service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:14.922007 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8d09d2-48f7-4f79-a0a0-635605afb414 service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:14.922279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8d09d2-48f7-4f79-a0a0-635605afb414 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:14.923876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-1b8d09d2-48f7-4f79-a0a0-635605afb414 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:14.965649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:15.016253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8d09d2-48f7-4f79-a0a0-635605afb414 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:55:15.017850 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8d09d2-48f7-4f79-a0a0-635605afb414 service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:55:15.036254 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 698/1388] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:55:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:55:15.194013 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c675065a-50d7-4a12-aa57-aa0aa4c4e8d8 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:15.194547 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c675065a-50d7-4a12-aa57-aa0aa4c4e8d8 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:15.194547 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c675065a-50d7-4a12-aa57-aa0aa4c4e8d8 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:15.196121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 691/1389] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:55:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:55:15.201304 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e295925-68c8-4bc7-beca-d1695a7e4c37 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:15.201781 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e295925-68c8-4bc7-beca-d1695a7e4c37 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:15.201871 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e295925-68c8-4bc7-beca-d1695a7e4c37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:15.277533 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-4e295925-68c8-4bc7-beca-d1695a7e4c37 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:15.278846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-4e295925-68c8-4bc7-beca-d1695a7e4c37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:15.403054 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 699/1390] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:55:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:15.707763 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9310ae3-b0b0-4c8d-9c86-fa18f727c4a1 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:15.708047 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9310ae3-b0b0-4c8d-9c86-fa18f727c4a1 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:15.708321 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9310ae3-b0b0-4c8d-9c86-fa18f727c4a1 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:15.709714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 692/1391] 10.222.0.62 () {54 vars in 962 bytes} [Fri Sep 23 13:55:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:55:15.715712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327d16a1-9fa9-4ab8-8949-49bc14f04199 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:15.716066 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327d16a1-9fa9-4ab8-8949-49bc14f04199 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:15.716404 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-327d16a1-9fa9-4ab8-8949-49bc14f04199 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:15.733044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:15.795401 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-327d16a1-9fa9-4ab8-8949-49bc14f04199 None None] MFA Rules not processed for user `acdf4a7bd608492390176dcc42ca1267`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:15.796611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-327d16a1-9fa9-4ab8-8949-49bc14f04199 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:15.873776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 700/1392] 10.222.0.62 () {58 vars in 1059 bytes} [Fri Sep 23 13:55:15 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:15.882880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9310ae3-b0b0-4c8d-9c86-fa18f727c4a1 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:55:15.884393 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f9310ae3-b0b0-4c8d-9c86-fa18f727c4a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:15.991478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9310ae3-b0b0-4c8d-9c86-fa18f727c4a1 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:55:16.056564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:16.067246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adfca00-7926-4031-804c-677555fb0a72 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'acdf4a7bd608492390176dcc42ca1267', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:55:16.067856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adfca00-7926-4031-804c-677555fb0a72 service glance] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:16.068124 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adfca00-7926-4031-804c-677555fb0a72 service glance] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:16.068387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adfca00-7926-4031-804c-677555fb0a72 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:16.070315 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3adfca00-7926-4031-804c-677555fb0a72 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:16.184362 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3adfca00-7926-4031-804c-677555fb0a72 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:55:16.185750 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3adfca00-7926-4031-804c-677555fb0a72 service glance] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:55:16.206144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 693/1393] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:55:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:55:17.125321 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ea849d-a7f6-455e-bb0a-8d2e6a1ba532 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:17.125816 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ea849d-a7f6-455e-bb0a-8d2e6a1ba532 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:17.125931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ea849d-a7f6-455e-bb0a-8d2e6a1ba532 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:17.193370 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-d5ea849d-a7f6-455e-bb0a-8d2e6a1ba532 None None] MFA Rules not processed for user `b471d768cd814ff8b4f6bd35d21e24a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:17.194632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d5ea849d-a7f6-455e-bb0a-8d2e6a1ba532 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:17.255469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:17.290697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 701/1394] 10.222.0.62 () {58 vars in 1009 bytes} [Fri Sep 23 13:55:17 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 13:55:17.335566 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12215bdd-98d6-4ddd-823a-459bb83476e3 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:17.336660 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12215bdd-98d6-4ddd-823a-459bb83476e3 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:17.336760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12215bdd-98d6-4ddd-823a-459bb83476e3 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:17.338359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 694/1395] 10.222.0.62 () {54 vars in 943 bytes} [Fri Sep 23 13:55:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:55:17.346314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057fa903-0571-4eec-921e-ac32a5fe1217 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:17.348056 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057fa903-0571-4eec-921e-ac32a5fe1217 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:17.348056 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057fa903-0571-4eec-921e-ac32a5fe1217 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:17.408677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8181afa3-68b1-4cd6-a8e3-8ca08baa9197 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:17.408947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8181afa3-68b1-4cd6-a8e3-8ca08baa9197 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:17.409228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8181afa3-68b1-4cd6-a8e3-8ca08baa9197 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:17.410607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 695/1396] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:55:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:55:17.416659 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e85c5d-6385-4a19-b4c1-3d5270aa198e None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:17.416938 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e85c5d-6385-4a19-b4c1-3d5270aa198e None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:17.417205 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e85c5d-6385-4a19-b4c1-3d5270aa198e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:17.426570 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-057fa903-0571-4eec-921e-ac32a5fe1217 None None] MFA Rules not processed for user `4b9269509f1d454eab7a3008c2ba0039`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:17.427790 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-057fa903-0571-4eec-921e-ac32a5fe1217 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:17.484472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-82e85c5d-6385-4a19-b4c1-3d5270aa198e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:17.485767 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-82e85c5d-6385-4a19-b4c1-3d5270aa198e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:17.528397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:17.536298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 702/1397] 10.222.0.62 () {58 vars in 1040 bytes} [Fri Sep 23 13:55:17 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:17.545883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057fa903-0571-4eec-921e-ac32a5fe1217 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:55:17.556143 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057fa903-0571-4eec-921e-ac32a5fe1217 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:55:17.582900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:17.585760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 696/1398] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:55:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 13:55:17.663636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62197b8-3c37-4408-bddf-624ae8d8f6d7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b9269509f1d454eab7a3008c2ba0039', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:55:17.665018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62197b8-3c37-4408-bddf-624ae8d8f6d7 service swift] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:17.665297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62197b8-3c37-4408-bddf-624ae8d8f6d7 service swift] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:17.665567 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62197b8-3c37-4408-bddf-624ae8d8f6d7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:17.667211 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c62197b8-3c37-4408-bddf-624ae8d8f6d7 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:17.749072 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62197b8-3c37-4408-bddf-624ae8d8f6d7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:55:17.750599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62197b8-3c37-4408-bddf-624ae8d8f6d7 service swift] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:55:17.758928 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 703/1399] 10.222.0.62 () {60 vars in 1384 bytes} [Fri Sep 23 13:55:17 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 23 13:55:17.873485 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e85c5d-6385-4a19-b4c1-3d5270aa198e None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:55:17.874754 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-82e85c5d-6385-4a19-b4c1-3d5270aa198e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:17.991009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e85c5d-6385-4a19-b4c1-3d5270aa198e None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:55:18.046413 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:18.106201 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d7d87b-71ae-462a-bd21-01be8ca9ea9f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:55:18.106860 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d7d87b-71ae-462a-bd21-01be8ca9ea9f service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:18.107139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d7d87b-71ae-462a-bd21-01be8ca9ea9f service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:18.107407 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d7d87b-71ae-462a-bd21-01be8ca9ea9f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:18.109003 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-44d7d87b-71ae-462a-bd21-01be8ca9ea9f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:18.200956 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d7d87b-71ae-462a-bd21-01be8ca9ea9f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:55:18.202587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d7d87b-71ae-462a-bd21-01be8ca9ea9f service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:55:18.223250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 697/1400] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:55:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:55:20.576343 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b7bc6d-3c1a-41e1-a90f-9eb8cce7b82b None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:20.576940 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b7bc6d-3c1a-41e1-a90f-9eb8cce7b82b None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:20.576940 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b7bc6d-3c1a-41e1-a90f-9eb8cce7b82b None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:20.578234 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 704/1401] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:55:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:55:20.583424 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef002db-6e23-4353-a6ad-158dddfd00cd None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:20.583898 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef002db-6e23-4353-a6ad-158dddfd00cd None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:20.584000 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef002db-6e23-4353-a6ad-158dddfd00cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:20.678805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-6ef002db-6e23-4353-a6ad-158dddfd00cd None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:20.681785 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-6ef002db-6e23-4353-a6ad-158dddfd00cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:20.696130 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:20.798944 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 698/1402] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:55:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:21.131586 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b7bc6d-3c1a-41e1-a90f-9eb8cce7b82b None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:55:21.133497 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-75b7bc6d-3c1a-41e1-a90f-9eb8cce7b82b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:21.213740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:21.349609 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b7bc6d-3c1a-41e1-a90f-9eb8cce7b82b None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:55:21.482891 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917765ec-1a8e-4682-9e17-9b62be647c16 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:55:21.483512 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917765ec-1a8e-4682-9e17-9b62be647c16 service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:21.483792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917765ec-1a8e-4682-9e17-9b62be647c16 service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:21.484061 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917765ec-1a8e-4682-9e17-9b62be647c16 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:21.485681 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-917765ec-1a8e-4682-9e17-9b62be647c16 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:21.595610 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-917765ec-1a8e-4682-9e17-9b62be647c16 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:55:21.598055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-917765ec-1a8e-4682-9e17-9b62be647c16 service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:55:21.616441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 705/1403] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:55:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 485 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:55:21.800796 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:23.985056 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49859ed-3039-4818-a606-07e66ba693f0 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:23.985630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49859ed-3039-4818-a606-07e66ba693f0 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:23.985630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49859ed-3039-4818-a606-07e66ba693f0 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:23.987356 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 699/1404] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:55:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:55:23.992978 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c651f141-fcab-482d-a5fd-61089cb39178 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:23.993379 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c651f141-fcab-482d-a5fd-61089cb39178 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:23.993542 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c651f141-fcab-482d-a5fd-61089cb39178 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:24.068038 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c651f141-fcab-482d-a5fd-61089cb39178 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:24.069240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c651f141-fcab-482d-a5fd-61089cb39178 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:24.176086 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:24.182009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 706/1405] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:55:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:24.477342 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49859ed-3039-4818-a606-07e66ba693f0 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:55:24.479203 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-b49859ed-3039-4818-a606-07e66ba693f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:24.635728 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:24.637854 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49859ed-3039-4818-a606-07e66ba693f0 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:55:24.763436 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb84a594-00b3-4ce1-8b74-e1938cb3dd50 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:55:24.764075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb84a594-00b3-4ce1-8b74-e1938cb3dd50 service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:24.764360 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb84a594-00b3-4ce1-8b74-e1938cb3dd50 service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:24.764633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb84a594-00b3-4ce1-8b74-e1938cb3dd50 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:24.766249 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-eb84a594-00b3-4ce1-8b74-e1938cb3dd50 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:24.864575 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb84a594-00b3-4ce1-8b74-e1938cb3dd50 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:55:24.866180 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb84a594-00b3-4ce1-8b74-e1938cb3dd50 service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:55:24.887468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 700/1406] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:55:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:55:27.252196 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a5fd3e-32f6-4277-80d8-4d9932e6e616 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:27.252794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a5fd3e-32f6-4277-80d8-4d9932e6e616 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:27.252794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a5fd3e-32f6-4277-80d8-4d9932e6e616 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:27.254097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 707/1407] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:55:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:55:27.259533 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8639f4c-0e1b-4be3-8094-2158fcc56218 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:27.265309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8639f4c-0e1b-4be3-8094-2158fcc56218 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:27.265309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8639f4c-0e1b-4be3-8094-2158fcc56218 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:27.265309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:27.341665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-a8639f4c-0e1b-4be3-8094-2158fcc56218 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:27.342899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a8639f4c-0e1b-4be3-8094-2158fcc56218 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:27.453541 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 701/1408] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:55:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:27.755971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a5fd3e-32f6-4277-80d8-4d9932e6e616 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:55:27.757566 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-88a5fd3e-32f6-4277-80d8-4d9932e6e616 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:27.918587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a5fd3e-32f6-4277-80d8-4d9932e6e616 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:55:27.992794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:28.045661 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b64a2c-55b1-4db2-81d5-414cfc795f83 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:55:28.046277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b64a2c-55b1-4db2-81d5-414cfc795f83 service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:28.046587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b64a2c-55b1-4db2-81d5-414cfc795f83 service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:28.046861 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b64a2c-55b1-4db2-81d5-414cfc795f83 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:28.048458 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-e1b64a2c-55b1-4db2-81d5-414cfc795f83 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:28.149892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b64a2c-55b1-4db2-81d5-414cfc795f83 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:55:28.151760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b64a2c-55b1-4db2-81d5-414cfc795f83 service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:55:28.172929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 708/1409] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:55:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:55:37.452038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=64 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:38.170110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=64 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:42.759046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfbf913-9f58-4069-843b-d749d079fa18 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:42.759639 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfbf913-9f58-4069-843b-d749d079fa18 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:42.759639 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfbf913-9f58-4069-843b-d749d079fa18 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:42.760922 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 702/1410] 10.222.0.62 () {54 vars in 918 bytes} [Fri Sep 23 13:55:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:55:42.765860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6acd614-139e-46e0-866e-347a8653551d None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:42.766299 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6acd614-139e-46e0-866e-347a8653551d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:42.766448 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6acd614-139e-46e0-866e-347a8653551d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:42.841768 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c6acd614-139e-46e0-866e-347a8653551d None None] MFA Rules not processed for user `ce249d4a99c34891b6ba941f57a14dd4`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:42.842997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c6acd614-139e-46e0-866e-347a8653551d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:42.948556 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 709/1411] 10.222.0.62 () {58 vars in 1015 bytes} [Fri Sep 23 13:55:42 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:42.963648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856b5be1-6457-48f7-8ef7-7be41835435e None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:42.963928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856b5be1-6457-48f7-8ef7-7be41835435e None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:42.964202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856b5be1-6457-48f7-8ef7-7be41835435e None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:42.965545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 703/1412] 10.222.0.62 () {54 vars in 918 bytes} [Fri Sep 23 13:55:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:55:42.970219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ec5f5d-01bf-4e22-acd5-a021d047bc66 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:42.970542 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ec5f5d-01bf-4e22-acd5-a021d047bc66 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:42.970820 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ec5f5d-01bf-4e22-acd5-a021d047bc66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:43.043905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:43.046596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-e9ec5f5d-01bf-4e22-acd5-a021d047bc66 None None] MFA Rules not processed for user `f01ed94dbb384a178ad0a45a6f3d8c17`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:43.047802 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-e9ec5f5d-01bf-4e22-acd5-a021d047bc66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:43.162419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 710/1413] 10.222.0.62 () {58 vars in 1015 bytes} [Fri Sep 23 13:55:42 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:45.738563 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79f4078-2ce9-411b-8849-ab6bddedd64a None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:45.739116 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79f4078-2ce9-411b-8849-ab6bddedd64a None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:45.739116 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79f4078-2ce9-411b-8849-ab6bddedd64a None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:45.740435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 704/1414] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 23 13:55:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:55:45.745792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981e300f-f7b6-452c-bb23-7b5629b13cd2 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:45.746195 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981e300f-f7b6-452c-bb23-7b5629b13cd2 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:45.746357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981e300f-f7b6-452c-bb23-7b5629b13cd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:45.811340 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-981e300f-f7b6-452c-bb23-7b5629b13cd2 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:45.812539 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-981e300f-f7b6-452c-bb23-7b5629b13cd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:45.866935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:45.908994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 711/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 23 13:55:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:46.203865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79f4078-2ce9-411b-8849-ab6bddedd64a None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:55:46.205430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a79f4078-2ce9-411b-8849-ab6bddedd64a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:46.351032 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79f4078-2ce9-411b-8849-ab6bddedd64a None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:55:46.424778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:46.465129 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312f261f-0f2b-4010-86d6-d517cb074ff3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:55:46.465737 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312f261f-0f2b-4010-86d6-d517cb074ff3 service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:46.466013 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312f261f-0f2b-4010-86d6-d517cb074ff3 service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:46.466282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312f261f-0f2b-4010-86d6-d517cb074ff3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:46.467955 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-312f261f-0f2b-4010-86d6-d517cb074ff3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:46.556669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312f261f-0f2b-4010-86d6-d517cb074ff3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:55:46.558263 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312f261f-0f2b-4010-86d6-d517cb074ff3 service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:55:46.575839 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 705/1416] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:55:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:55:47.723279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf957fd-c9e4-415a-adeb-8f4213a3da14 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:47.723818 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf957fd-c9e4-415a-adeb-8f4213a3da14 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:47.723937 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf957fd-c9e4-415a-adeb-8f4213a3da14 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:47.725213 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 712/1417] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 23 13:55:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:55:47.732017 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27056a90-2ffb-4796-b79a-78fe0742025b None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:47.732468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27056a90-2ffb-4796-b79a-78fe0742025b None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:47.732588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27056a90-2ffb-4796-b79a-78fe0742025b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:47.807656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-27056a90-2ffb-4796-b79a-78fe0742025b None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:47.808850 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-27056a90-2ffb-4796-b79a-78fe0742025b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:47.834652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:47.901621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 706/1418] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 23 13:55:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:48.056314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf957fd-c9e4-415a-adeb-8f4213a3da14 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:55:48.057820 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-abf957fd-c9e4-415a-adeb-8f4213a3da14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:48.200854 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf957fd-c9e4-415a-adeb-8f4213a3da14 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:55:48.240816 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:48.312562 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ae73ab-307c-48d5-ac71-070e65f102f4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:55:48.313155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ae73ab-307c-48d5-ac71-070e65f102f4 service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:48.313430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ae73ab-307c-48d5-ac71-070e65f102f4 service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:48.313696 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ae73ab-307c-48d5-ac71-070e65f102f4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:48.315300 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c3ae73ab-307c-48d5-ac71-070e65f102f4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:48.403710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ae73ab-307c-48d5-ac71-070e65f102f4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:55:48.405282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ae73ab-307c-48d5-ac71-070e65f102f4 service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:55:48.426200 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 713/1419] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:55:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:55:49.546905 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff235aab-a8b2-49c4-9238-d9744369ad91 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:49.547435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff235aab-a8b2-49c4-9238-d9744369ad91 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:49.547435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff235aab-a8b2-49c4-9238-d9744369ad91 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:49.548753 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 707/1420] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 23 13:55:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:55:49.554949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66398522-4293-4c3c-bc23-c10d49d72c2c None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:49.555356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66398522-4293-4c3c-bc23-c10d49d72c2c None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:49.555482 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66398522-4293-4c3c-bc23-c10d49d72c2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:49.609134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:49.630024 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-66398522-4293-4c3c-bc23-c10d49d72c2c None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:49.631245 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-66398522-4293-4c3c-bc23-c10d49d72c2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:49.727357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 714/1421] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 23 13:55:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:49.880276 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff235aab-a8b2-49c4-9238-d9744369ad91 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:55:49.881714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ff235aab-a8b2-49c4-9238-d9744369ad91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:50.032266 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff235aab-a8b2-49c4-9238-d9744369ad91 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:55:50.045345 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:50.146391 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e91bceb-29cc-4172-a045-b9bd28ea946a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:55:50.146940 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e91bceb-29cc-4172-a045-b9bd28ea946a service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:50.147221 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e91bceb-29cc-4172-a045-b9bd28ea946a service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:50.147487 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e91bceb-29cc-4172-a045-b9bd28ea946a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:50.149763 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-6e91bceb-29cc-4172-a045-b9bd28ea946a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:50.362521 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e91bceb-29cc-4172-a045-b9bd28ea946a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:55:50.364111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e91bceb-29cc-4172-a045-b9bd28ea946a service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:55:50.383925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 708/1422] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:55:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 504 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:55:51.495904 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941d6c31-c46c-4262-98ad-8ed1ed40ce96 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:51.496420 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941d6c31-c46c-4262-98ad-8ed1ed40ce96 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:51.496420 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941d6c31-c46c-4262-98ad-8ed1ed40ce96 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:51.497813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 715/1423] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 23 13:55:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:55:51.504041 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1de76f-5d32-4c65-ac7c-ce67de56e57c None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:51.504456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1de76f-5d32-4c65-ac7c-ce67de56e57c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:51.504549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1de76f-5d32-4c65-ac7c-ce67de56e57c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:51.525272 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:51.578620 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-dc1de76f-5d32-4c65-ac7c-ce67de56e57c None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:51.579807 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-dc1de76f-5d32-4c65-ac7c-ce67de56e57c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:51.677057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 709/1424] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 23 13:55:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:51.827379 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941d6c31-c46c-4262-98ad-8ed1ed40ce96 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:55:51.828805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-941d6c31-c46c-4262-98ad-8ed1ed40ce96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:51.940261 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:51.978925 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941d6c31-c46c-4262-98ad-8ed1ed40ce96 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:55:52.094518 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965c62bb-8c0f-499e-8b01-9381bde0defc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:55:52.095041 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965c62bb-8c0f-499e-8b01-9381bde0defc service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:52.095314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965c62bb-8c0f-499e-8b01-9381bde0defc service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:52.095578 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965c62bb-8c0f-499e-8b01-9381bde0defc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:52.097178 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-965c62bb-8c0f-499e-8b01-9381bde0defc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:52.174936 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:52.188360 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965c62bb-8c0f-499e-8b01-9381bde0defc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:55:52.190007 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965c62bb-8c0f-499e-8b01-9381bde0defc service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:55:52.208639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 716/1425] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:55:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:55:53.325161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d807b0-b54c-4f55-b69d-11c4c0378ad2 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:53.325686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d807b0-b54c-4f55-b69d-11c4c0378ad2 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:53.325810 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d807b0-b54c-4f55-b69d-11c4c0378ad2 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:53.327170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 710/1426] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 23 13:55:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:55:53.333906 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c5ba8b-7a5b-4dbb-ad60-5904c7dae9da None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:53.337161 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c5ba8b-7a5b-4dbb-ad60-5904c7dae9da None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:53.337161 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c5ba8b-7a5b-4dbb-ad60-5904c7dae9da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:53.410677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-f5c5ba8b-7a5b-4dbb-ad60-5904c7dae9da None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:53.411887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f5c5ba8b-7a5b-4dbb-ad60-5904c7dae9da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:53.511005 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 717/1427] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 23 13:55:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:53.666921 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d807b0-b54c-4f55-b69d-11c4c0378ad2 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:55:53.668435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f9d807b0-b54c-4f55-b69d-11c4c0378ad2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:53.751296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:53.812446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d807b0-b54c-4f55-b69d-11c4c0378ad2 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:55:53.923584 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a00354-7f59-4bdc-b8fc-b4df797d6ac4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:55:53.924172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a00354-7f59-4bdc-b8fc-b4df797d6ac4 service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:53.924444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a00354-7f59-4bdc-b8fc-b4df797d6ac4 service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:53.924705 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a00354-7f59-4bdc-b8fc-b4df797d6ac4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:53.926257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e6a00354-7f59-4bdc-b8fc-b4df797d6ac4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:53.973383 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:54.013215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a00354-7f59-4bdc-b8fc-b4df797d6ac4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:55:54.014816 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a00354-7f59-4bdc-b8fc-b4df797d6ac4 service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:55:54.032506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 711/1428] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:55:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:55:55.161479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c2b812-e090-4687-b951-5e056cb86b02 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:55.162060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c2b812-e090-4687-b951-5e056cb86b02 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:55.162060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c2b812-e090-4687-b951-5e056cb86b02 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:55.163489 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 718/1429] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 23 13:55:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:55:55.169775 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf20234-1de5-4c0e-ad30-90c7761b1703 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:55.170203 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf20234-1de5-4c0e-ad30-90c7761b1703 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:55.171767 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf20234-1de5-4c0e-ad30-90c7761b1703 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:55.248420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-4cf20234-1de5-4c0e-ad30-90c7761b1703 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:55.249611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-4cf20234-1de5-4c0e-ad30-90c7761b1703 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:55.345737 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 712/1430] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 23 13:55:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:55.499372 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c2b812-e090-4687-b951-5e056cb86b02 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:55:55.500897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-56c2b812-e090-4687-b951-5e056cb86b02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:55.553565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:55.662461 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c2b812-e090-4687-b951-5e056cb86b02 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:55:55.787749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6affba8e-8cb2-45dc-98ef-6fe24c47346e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:55:55.789100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6affba8e-8cb2-45dc-98ef-6fe24c47346e service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:55.789385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6affba8e-8cb2-45dc-98ef-6fe24c47346e service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:55.789653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6affba8e-8cb2-45dc-98ef-6fe24c47346e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:55.791306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-6affba8e-8cb2-45dc-98ef-6fe24c47346e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:55.798619 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:55.897312 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6affba8e-8cb2-45dc-98ef-6fe24c47346e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:55:55.900755 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6affba8e-8cb2-45dc-98ef-6fe24c47346e service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:55:55.919988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 719/1431] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:55:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:55:57.045903 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5735caec-d1ff-4ee6-8f20-18f8c7bb5d26 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:57.046474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5735caec-d1ff-4ee6-8f20-18f8c7bb5d26 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:57.046474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5735caec-d1ff-4ee6-8f20-18f8c7bb5d26 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:57.047815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 713/1432] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 23 13:55:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:55:57.054567 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a40b0db-3731-4f43-a09e-7afb9762a9df None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:57.054567 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a40b0db-3731-4f43-a09e-7afb9762a9df None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:57.055434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a40b0db-3731-4f43-a09e-7afb9762a9df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:57.129569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-8a40b0db-3731-4f43-a09e-7afb9762a9df None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:57.130807 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8a40b0db-3731-4f43-a09e-7afb9762a9df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:57.197970 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:57.226570 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 720/1433] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 23 13:55:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:57.377801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5735caec-d1ff-4ee6-8f20-18f8c7bb5d26 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:55:57.379271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-5735caec-d1ff-4ee6-8f20-18f8c7bb5d26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:57.396449 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:57.523538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5735caec-d1ff-4ee6-8f20-18f8c7bb5d26 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:55:57.612269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:57.638129 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6928f8-ad98-4dd9-a34f-cc67685062ac service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:55:57.638698 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6928f8-ad98-4dd9-a34f-cc67685062ac service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:57.638975 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6928f8-ad98-4dd9-a34f-cc67685062ac service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:57.639247 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6928f8-ad98-4dd9-a34f-cc67685062ac service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:57.640840 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-6b6928f8-ad98-4dd9-a34f-cc67685062ac service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:57.729147 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6928f8-ad98-4dd9-a34f-cc67685062ac service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:55:57.730736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6928f8-ad98-4dd9-a34f-cc67685062ac service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:55:57.747911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 714/1434] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:55:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:55:58.869389 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d28b4a2-139a-4a81-bb63-a5c6631d6ef0 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:58.869959 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d28b4a2-139a-4a81-bb63-a5c6631d6ef0 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:58.869959 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d28b4a2-139a-4a81-bb63-a5c6631d6ef0 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:58.871351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 721/1435] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 23 13:55:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:55:58.877581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc0cf4f-d895-4669-8d09-567bfba290ed None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:58.878022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc0cf4f-d895-4669-8d09-567bfba290ed None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:58.878120 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc0cf4f-d895-4669-8d09-567bfba290ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:58.953119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-5dc0cf4f-d895-4669-8d09-567bfba290ed None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:55:58.954353 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-5dc0cf4f-d895-4669-8d09-567bfba290ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:58.995259 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:59.048979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 715/1436] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 23 13:55:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:55:59.199209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d28b4a2-139a-4a81-bb63-a5c6631d6ef0 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:55:59.201018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8d28b4a2-139a-4a81-bb63-a5c6631d6ef0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:59.346204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d28b4a2-139a-4a81-bb63-a5c6631d6ef0 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:55:59.400785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:55:59.458500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d89624e-7bd8-4fa5-ac10-643fb1838336 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:55:59.459100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d89624e-7bd8-4fa5-ac10-643fb1838336 service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:55:59.459374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d89624e-7bd8-4fa5-ac10-643fb1838336 service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:55:59.459640 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d89624e-7bd8-4fa5-ac10-643fb1838336 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:55:59.461599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8d89624e-7bd8-4fa5-ac10-643fb1838336 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:55:59.548977 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d89624e-7bd8-4fa5-ac10-643fb1838336 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:55:59.550596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d89624e-7bd8-4fa5-ac10-643fb1838336 service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:55:59.569301 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 722/1437] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:55:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:56:00.679829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432ba97f-1b8a-4005-8fe6-b9cc7f878ac4 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:00.680450 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432ba97f-1b8a-4005-8fe6-b9cc7f878ac4 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:00.680450 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432ba97f-1b8a-4005-8fe6-b9cc7f878ac4 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:00.681727 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 716/1438] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 23 13:56:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:56:00.687927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81b877e-7fb6-485e-989b-3c3beaf9e600 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:00.688374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81b877e-7fb6-485e-989b-3c3beaf9e600 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:00.688478 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81b877e-7fb6-485e-989b-3c3beaf9e600 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:00.764386 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-b81b877e-7fb6-485e-989b-3c3beaf9e600 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:56:00.765589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-b81b877e-7fb6-485e-989b-3c3beaf9e600 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:00.767652 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:00.861002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 723/1439] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 23 13:56:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 13:56:01.012625 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432ba97f-1b8a-4005-8fe6-b9cc7f878ac4 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:56:01.014128 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-432ba97f-1b8a-4005-8fe6-b9cc7f878ac4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:01.155654 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432ba97f-1b8a-4005-8fe6-b9cc7f878ac4 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:56:01.184015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:01.273575 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1edd1ab-a872-4eb7-99d2-0018f8924314 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:56:01.274177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1edd1ab-a872-4eb7-99d2-0018f8924314 service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:01.274475 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1edd1ab-a872-4eb7-99d2-0018f8924314 service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:01.274733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1edd1ab-a872-4eb7-99d2-0018f8924314 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:01.276300 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f1edd1ab-a872-4eb7-99d2-0018f8924314 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:01.363298 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1edd1ab-a872-4eb7-99d2-0018f8924314 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:56:01.364862 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1edd1ab-a872-4eb7-99d2-0018f8924314 service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:56:01.382939 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 717/1440] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:56:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:56:03.459377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5c9b96-6d0f-4920-b71e-5e7b684f1c86 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:03.459963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5c9b96-6d0f-4920-b71e-5e7b684f1c86 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:03.459963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5c9b96-6d0f-4920-b71e-5e7b684f1c86 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:03.462077 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 724/1441] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 23 13:56:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:56:03.467446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8617bc16-5ecc-42f6-8458-659187eb8bc3 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:03.467892 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8617bc16-5ecc-42f6-8458-659187eb8bc3 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:03.468003 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8617bc16-5ecc-42f6-8458-659187eb8bc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:03.507102 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:03.532997 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-8617bc16-5ecc-42f6-8458-659187eb8bc3 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:56:03.534186 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8617bc16-5ecc-42f6-8458-659187eb8bc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:03.627836 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 718/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 23 13:56:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:56:03.920541 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5c9b96-6d0f-4920-b71e-5e7b684f1c86 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:56:03.922083 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-5b5c9b96-6d0f-4920-b71e-5e7b684f1c86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:04.026550 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5c9b96-6d0f-4920-b71e-5e7b684f1c86 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:56:04.059532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:04.100428 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8faefe1-4771-4a02-868a-016befbafe75 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:56:04.101060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8faefe1-4771-4a02-868a-016befbafe75 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:04.101481 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8faefe1-4771-4a02-868a-016befbafe75 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:04.101797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8faefe1-4771-4a02-868a-016befbafe75 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:04.103395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-b8faefe1-4771-4a02-868a-016befbafe75 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:04.209687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8faefe1-4771-4a02-868a-016befbafe75 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:56:04.212119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8faefe1-4771-4a02-868a-016befbafe75 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:56:04.230815 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 725/1443] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:56:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:56:06.548043 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd968d3f-e9bc-4ea4-9df6-78ea4d29a4ce None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:06.548579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd968d3f-e9bc-4ea4-9df6-78ea4d29a4ce None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:06.548579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd968d3f-e9bc-4ea4-9df6-78ea4d29a4ce None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:06.549835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 719/1444] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 23 13:56:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:56:06.555787 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d097304a-a59c-451d-9410-0eefd3016592 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:06.556237 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d097304a-a59c-451d-9410-0eefd3016592 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:06.556345 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d097304a-a59c-451d-9410-0eefd3016592 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:06.621485 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-d097304a-a59c-451d-9410-0eefd3016592 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:56:06.622700 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d097304a-a59c-451d-9410-0eefd3016592 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:06.635349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:06.717780 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 726/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 23 13:56:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:56:07.002262 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd968d3f-e9bc-4ea4-9df6-78ea4d29a4ce None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:56:07.003820 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-bd968d3f-e9bc-4ea4-9df6-78ea4d29a4ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:07.111279 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd968d3f-e9bc-4ea4-9df6-78ea4d29a4ce None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:56:07.125386 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:07.189512 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ac915f-c9cf-4e1c-b076-1ac0e6032455 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:56:07.190155 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ac915f-c9cf-4e1c-b076-1ac0e6032455 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:07.190469 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ac915f-c9cf-4e1c-b076-1ac0e6032455 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:07.190742 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ac915f-c9cf-4e1c-b076-1ac0e6032455 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:07.192332 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-95ac915f-c9cf-4e1c-b076-1ac0e6032455 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:07.306064 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ac915f-c9cf-4e1c-b076-1ac0e6032455 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:56:07.307510 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ac915f-c9cf-4e1c-b076-1ac0e6032455 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:56:07.325834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 720/1446] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:56:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:56:09.808067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89180712-8e8b-447e-bcce-440a914d501d None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:09.808623 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89180712-8e8b-447e-bcce-440a914d501d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:09.808623 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89180712-8e8b-447e-bcce-440a914d501d None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:09.809947 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 727/1447] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 23 13:56:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:56:09.815362 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfa0b8e-16f3-4219-b096-69540f750d48 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:09.815756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfa0b8e-16f3-4219-b096-69540f750d48 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:09.815901 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfa0b8e-16f3-4219-b096-69540f750d48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:09.863150 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:09.880476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-8cfa0b8e-16f3-4219-b096-69540f750d48 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:56:09.881689 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8cfa0b8e-16f3-4219-b096-69540f750d48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:09.979570 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 721/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 23 13:56:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:56:10.265660 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89180712-8e8b-447e-bcce-440a914d501d None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:56:10.267249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-89180712-8e8b-447e-bcce-440a914d501d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:10.371670 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89180712-8e8b-447e-bcce-440a914d501d None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:56:10.412671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:10.447082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0c4709-5e3e-4933-894d-49604baa4d79 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:56:10.447682 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0c4709-5e3e-4933-894d-49604baa4d79 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:10.447953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0c4709-5e3e-4933-894d-49604baa4d79 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:10.448216 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0c4709-5e3e-4933-894d-49604baa4d79 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:10.449783 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8d0c4709-5e3e-4933-894d-49604baa4d79 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:10.560389 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0c4709-5e3e-4933-894d-49604baa4d79 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:56:10.561794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0c4709-5e3e-4933-894d-49604baa4d79 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:56:10.580356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 728/1449] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:56:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:56:12.854414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df21864-1d2d-4bbb-8562-b72af0ecf551 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:12.854980 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df21864-1d2d-4bbb-8562-b72af0ecf551 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:12.854980 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df21864-1d2d-4bbb-8562-b72af0ecf551 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:12.856299 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 722/1450] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 23 13:56:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:56:12.861556 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4a7987-f94f-4267-8f0a-d81ebb098893 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:12.861953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4a7987-f94f-4267-8f0a-d81ebb098893 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:12.862097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4a7987-f94f-4267-8f0a-d81ebb098893 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:12.927033 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-7b4a7987-f94f-4267-8f0a-d81ebb098893 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:56:12.928249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-7b4a7987-f94f-4267-8f0a-d81ebb098893 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:12.949283 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:13.029476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 729/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 23 13:56:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:56:13.314898 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df21864-1d2d-4bbb-8562-b72af0ecf551 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:56:13.316466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2df21864-1d2d-4bbb-8562-b72af0ecf551 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:13.418585 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df21864-1d2d-4bbb-8562-b72af0ecf551 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:56:13.441437 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:13.495255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2d4d7a-07b5-48bf-9056-ec4edc5364b8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:56:13.495793 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2d4d7a-07b5-48bf-9056-ec4edc5364b8 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:13.496067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2d4d7a-07b5-48bf-9056-ec4edc5364b8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:13.496336 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2d4d7a-07b5-48bf-9056-ec4edc5364b8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:13.497917 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-5c2d4d7a-07b5-48bf-9056-ec4edc5364b8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:13.605698 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c2d4d7a-07b5-48bf-9056-ec4edc5364b8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:56:13.607149 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c2d4d7a-07b5-48bf-9056-ec4edc5364b8 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:56:13.625407 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 723/1452] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:56:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:56:15.835770 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0947b671-8bd0-4ccd-92f1-5612236c9e80 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:15.836331 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0947b671-8bd0-4ccd-92f1-5612236c9e80 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:15.836331 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0947b671-8bd0-4ccd-92f1-5612236c9e80 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:15.837647 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 730/1453] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 23 13:56:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:56:15.843007 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95dc99c-5b38-4031-96e0-28c8adb83593 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:15.843418 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95dc99c-5b38-4031-96e0-28c8adb83593 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:15.843602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95dc99c-5b38-4031-96e0-28c8adb83593 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:15.907325 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:15.909953 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-d95dc99c-5b38-4031-96e0-28c8adb83593 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:56:15.911204 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d95dc99c-5b38-4031-96e0-28c8adb83593 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:16.008870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 724/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 23 13:56:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:56:16.294125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0947b671-8bd0-4ccd-92f1-5612236c9e80 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:56:16.295743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-0947b671-8bd0-4ccd-92f1-5612236c9e80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:16.399281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0947b671-8bd0-4ccd-92f1-5612236c9e80 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:56:16.448842 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:16.472987 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf20c67-323b-483c-b256-b4b955f6ee2c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:56:16.473563 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf20c67-323b-483c-b256-b4b955f6ee2c service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:16.473839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf20c67-323b-483c-b256-b4b955f6ee2c service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:16.474106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf20c67-323b-483c-b256-b4b955f6ee2c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:16.475719 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-aaf20c67-323b-483c-b256-b4b955f6ee2c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:16.704319 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf20c67-323b-483c-b256-b4b955f6ee2c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:56:16.705804 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf20c67-323b-483c-b256-b4b955f6ee2c service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:56:16.724117 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 731/1455] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:56:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:56:19.001791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd6c20c-dbc2-465c-88d2-3712b2b05ad0 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:19.002417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd6c20c-dbc2-465c-88d2-3712b2b05ad0 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:19.002417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd6c20c-dbc2-465c-88d2-3712b2b05ad0 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:19.003721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 725/1456] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 23 13:56:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:56:19.010269 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ae7928-43d3-4d37-8439-63b16e002a57 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:19.010774 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ae7928-43d3-4d37-8439-63b16e002a57 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:19.010878 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ae7928-43d3-4d37-8439-63b16e002a57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:19.075821 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-a2ae7928-43d3-4d37-8439-63b16e002a57 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:56:19.077006 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a2ae7928-43d3-4d37-8439-63b16e002a57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:19.105809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:19.172180 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 732/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 23 13:56:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:56:19.453794 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd6c20c-dbc2-465c-88d2-3712b2b05ad0 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:56:19.455414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9bd6c20c-dbc2-465c-88d2-3712b2b05ad0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:19.557785 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd6c20c-dbc2-465c-88d2-3712b2b05ad0 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:56:19.586255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:19.630876 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45f8559-fe1f-4a39-b9c5-5f7d7b4fa773 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:56:19.631717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45f8559-fe1f-4a39-b9c5-5f7d7b4fa773 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:19.631992 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45f8559-fe1f-4a39-b9c5-5f7d7b4fa773 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:19.632265 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45f8559-fe1f-4a39-b9c5-5f7d7b4fa773 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:19.633840 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d45f8559-fe1f-4a39-b9c5-5f7d7b4fa773 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:19.742670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45f8559-fe1f-4a39-b9c5-5f7d7b4fa773 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:56:19.744094 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45f8559-fe1f-4a39-b9c5-5f7d7b4fa773 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:56:19.762230 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 726/1458] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:56:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:56:21.973043 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0608d615-52d4-4bac-aca5-f74cca723e56 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:21.973569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0608d615-52d4-4bac-aca5-f74cca723e56 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:21.973569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0608d615-52d4-4bac-aca5-f74cca723e56 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:21.975165 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 733/1459] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 23 13:56:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:56:21.980580 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330b3a46-1f20-4d17-8cf9-d9942227aeb3 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:21.981002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330b3a46-1f20-4d17-8cf9-d9942227aeb3 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:21.981233 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330b3a46-1f20-4d17-8cf9-d9942227aeb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:22.048387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-330b3a46-1f20-4d17-8cf9-d9942227aeb3 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:56:22.049606 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-330b3a46-1f20-4d17-8cf9-d9942227aeb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:22.058848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:22.147919 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 727/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 23 13:56:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:56:22.435015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0608d615-52d4-4bac-aca5-f74cca723e56 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:56:22.436562 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-0608d615-52d4-4bac-aca5-f74cca723e56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:22.539071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0608d615-52d4-4bac-aca5-f74cca723e56 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:56:22.593622 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:22.610543 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee7c417-a57d-42c6-8220-fc38c5ff48d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:56:22.611143 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee7c417-a57d-42c6-8220-fc38c5ff48d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:22.611420 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee7c417-a57d-42c6-8220-fc38c5ff48d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:22.611686 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee7c417-a57d-42c6-8220-fc38c5ff48d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:22.613248 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-3ee7c417-a57d-42c6-8220-fc38c5ff48d4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:22.721238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee7c417-a57d-42c6-8220-fc38c5ff48d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:56:22.722688 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee7c417-a57d-42c6-8220-fc38c5ff48d4 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:56:22.740136 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 734/1461] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:56:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:56:24.960060 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9612cab8-e889-4978-8f10-422e60d34535 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:24.960652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9612cab8-e889-4978-8f10-422e60d34535 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:24.960652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9612cab8-e889-4978-8f10-422e60d34535 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:24.961954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 728/1462] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 23 13:56:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:56:24.967251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39303bda-0344-4152-8c2b-814fdc277d1b None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:24.967681 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39303bda-0344-4152-8c2b-814fdc277d1b None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:24.967787 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39303bda-0344-4152-8c2b-814fdc277d1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:25.033071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-39303bda-0344-4152-8c2b-814fdc277d1b None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:56:25.034260 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-39303bda-0344-4152-8c2b-814fdc277d1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:25.072012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:25.128895 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 735/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 23 13:56:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:56:25.421064 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9612cab8-e889-4978-8f10-422e60d34535 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:56:25.422658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9612cab8-e889-4978-8f10-422e60d34535 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:25.525190 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9612cab8-e889-4978-8f10-422e60d34535 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:56:25.562108 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:25.597025 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ffdb31-8363-4e58-8e3e-eabfbe1d0b90 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:56:25.597619 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ffdb31-8363-4e58-8e3e-eabfbe1d0b90 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:25.597894 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ffdb31-8363-4e58-8e3e-eabfbe1d0b90 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:25.598162 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ffdb31-8363-4e58-8e3e-eabfbe1d0b90 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:25.599915 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-21ffdb31-8363-4e58-8e3e-eabfbe1d0b90 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:25.707788 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ffdb31-8363-4e58-8e3e-eabfbe1d0b90 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:56:25.709647 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ffdb31-8363-4e58-8e3e-eabfbe1d0b90 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:56:25.729038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 729/1464] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:56:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:56:27.920018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc680b0a-2c34-4596-b753-4048c30116f9 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:27.920554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc680b0a-2c34-4596-b753-4048c30116f9 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:27.920554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc680b0a-2c34-4596-b753-4048c30116f9 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:27.921829 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 736/1465] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 23 13:56:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:56:27.927052 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6fdeb2-fb61-4225-8f59-82a4ce8e1283 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:27.928079 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6fdeb2-fb61-4225-8f59-82a4ce8e1283 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:27.928354 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6fdeb2-fb61-4225-8f59-82a4ce8e1283 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:27.992896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-db6fdeb2-fb61-4225-8f59-82a4ce8e1283 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:56:27.994084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-db6fdeb2-fb61-4225-8f59-82a4ce8e1283 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:28.012173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:28.089594 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 730/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 23 13:56:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:56:28.374516 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc680b0a-2c34-4596-b753-4048c30116f9 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:56:28.376140 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-dc680b0a-2c34-4596-b753-4048c30116f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:28.480730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc680b0a-2c34-4596-b753-4048c30116f9 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:56:28.544720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:28.550785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d43c39-ec19-473b-91e1-81157d977502 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:56:28.551378 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d43c39-ec19-473b-91e1-81157d977502 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:28.551653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d43c39-ec19-473b-91e1-81157d977502 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:28.551920 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d43c39-ec19-473b-91e1-81157d977502 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:28.553611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-86d43c39-ec19-473b-91e1-81157d977502 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:28.658792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d43c39-ec19-473b-91e1-81157d977502 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:56:28.660192 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d43c39-ec19-473b-91e1-81157d977502 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:56:28.678835 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 737/1467] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:56:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:56:30.844667 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accae1bf-8ffe-4594-991d-d717b0cdb413 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:30.845234 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accae1bf-8ffe-4594-991d-d717b0cdb413 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:30.845234 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-accae1bf-8ffe-4594-991d-d717b0cdb413 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:30.846582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 731/1468] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 23 13:56:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:56:30.852213 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02a8388-36ab-44c1-8c12-01a0c553b686 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:30.852620 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02a8388-36ab-44c1-8c12-01a0c553b686 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:30.852763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02a8388-36ab-44c1-8c12-01a0c553b686 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:30.916763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-d02a8388-36ab-44c1-8c12-01a0c553b686 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:56:30.917947 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d02a8388-36ab-44c1-8c12-01a0c553b686 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:30.965162 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:31.013122 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 738/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 23 13:56:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:56:31.298723 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-accae1bf-8ffe-4594-991d-d717b0cdb413 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:56:31.300238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-accae1bf-8ffe-4594-991d-d717b0cdb413 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:31.406728 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-accae1bf-8ffe-4594-991d-d717b0cdb413 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:56:31.454910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:31.481370 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6005fb-894c-415b-b98f-e264de0b1911 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:56:31.481905 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6005fb-894c-415b-b98f-e264de0b1911 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:31.482181 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6005fb-894c-415b-b98f-e264de0b1911 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:31.482487 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6005fb-894c-415b-b98f-e264de0b1911 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:31.484072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9b6005fb-894c-415b-b98f-e264de0b1911 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:31.592311 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6005fb-894c-415b-b98f-e264de0b1911 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:56:31.593726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6005fb-894c-415b-b98f-e264de0b1911 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:56:31.612182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 732/1470] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:56:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:56:41.011621 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=22 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:41.609889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=66 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:51.693990 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56307f4e-d699-499c-88bd-02adff536461 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:51.694496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56307f4e-d699-499c-88bd-02adff536461 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:51.694605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56307f4e-d699-499c-88bd-02adff536461 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:51.696001 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 739/1471] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:56:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:56:51.701409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ba36b6-86eb-4a3f-a4a6-e47eff0e9ea0 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:51.701819 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ba36b6-86eb-4a3f-a4a6-e47eff0e9ea0 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:51.701947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ba36b6-86eb-4a3f-a4a6-e47eff0e9ea0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:51.768892 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-b0ba36b6-86eb-4a3f-a4a6-e47eff0e9ea0 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:56:51.770767 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-b0ba36b6-86eb-4a3f-a4a6-e47eff0e9ea0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:51.868779 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 733/1472] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:56:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:56:52.171395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25795350-ce09-4da3-8ffc-d1b50903f259 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:52.171679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25795350-ce09-4da3-8ffc-d1b50903f259 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:52.171950 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25795350-ce09-4da3-8ffc-d1b50903f259 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:52.173327 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 740/1473] 10.222.0.62 () {54 vars in 962 bytes} [Fri Sep 23 13:56:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:56:52.178920 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42794a9e-c81b-4098-8c2a-4577a898e8f7 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:52.179205 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42794a9e-c81b-4098-8c2a-4577a898e8f7 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:52.179506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42794a9e-c81b-4098-8c2a-4577a898e8f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:52.256641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-42794a9e-c81b-4098-8c2a-4577a898e8f7 None None] MFA Rules not processed for user `acdf4a7bd608492390176dcc42ca1267`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:56:52.257831 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-42794a9e-c81b-4098-8c2a-4577a898e8f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:52.277787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:52.332433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 734/1474] 10.222.0.62 () {58 vars in 1059 bytes} [Fri Sep 23 13:56:52 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:56:52.340673 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25795350-ce09-4da3-8ffc-d1b50903f259 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:56:52.342411 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-25795350-ce09-4da3-8ffc-d1b50903f259 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:52.448395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25795350-ce09-4da3-8ffc-d1b50903f259 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:56:52.521181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde1e7d0-d7cc-4a07-96a8-87e8ca9ef63c service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'acdf4a7bd608492390176dcc42ca1267', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:56:52.521701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde1e7d0-d7cc-4a07-96a8-87e8ca9ef63c service glance] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:52.521979 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde1e7d0-d7cc-4a07-96a8-87e8ca9ef63c service glance] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:52.522250 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde1e7d0-d7cc-4a07-96a8-87e8ca9ef63c service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:52.523863 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-bde1e7d0-d7cc-4a07-96a8-87e8ca9ef63c service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:52.576419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:52.632735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde1e7d0-d7cc-4a07-96a8-87e8ca9ef63c service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:56:52.634141 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde1e7d0-d7cc-4a07-96a8-87e8ca9ef63c service glance] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:56:52.651934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 741/1475] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:56:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:56:54.903360 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55689f5-8ae2-4f5f-aa71-91dd83a7a769 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:54.903954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55689f5-8ae2-4f5f-aa71-91dd83a7a769 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:54.903954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55689f5-8ae2-4f5f-aa71-91dd83a7a769 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:54.905321 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 735/1476] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:56:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:56:54.911224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3d6cb3-821c-46ed-ac95-3bb05dbbc12c None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:54.911624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3d6cb3-821c-46ed-ac95-3bb05dbbc12c None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:54.911764 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3d6cb3-821c-46ed-ac95-3bb05dbbc12c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:54.977131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-9b3d6cb3-821c-46ed-ac95-3bb05dbbc12c None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:56:54.978338 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-9b3d6cb3-821c-46ed-ac95-3bb05dbbc12c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:55.072825 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 742/1477] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:56:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:56:55.351491 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55689f5-8ae2-4f5f-aa71-91dd83a7a769 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:56:55.353008 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a55689f5-8ae2-4f5f-aa71-91dd83a7a769 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:55.480595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a55689f5-8ae2-4f5f-aa71-91dd83a7a769 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:56:55.530720 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:55.574264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f2aef1-200f-4691-8308-e1b437f41996 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:56:55.574896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f2aef1-200f-4691-8308-e1b437f41996 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:55.575173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f2aef1-200f-4691-8308-e1b437f41996 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:55.575441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f2aef1-200f-4691-8308-e1b437f41996 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:55.576097 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f2aef1-200f-4691-8308-e1b437f41996 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:56:55.577844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: WARNING py.warnings [None req-00f2aef1-200f-4691-8308-e1b437f41996 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 23 13:56:55.577844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: warnings.warn(msg) Sep 23 13:56:55.577844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: Sep 23 13:56:55.579007 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f2aef1-200f-4691-8308-e1b437f41996 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:56:55.597955 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 736/1478] 10.222.0.62 () {56 vars in 1108 bytes} [Fri Sep 23 13:56:55 2022] GET /identity/v3/projects => generated 2553 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 13:56:57.868165 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c82f8da-5194-4e61-962d-8178ae088455 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:57.868737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c82f8da-5194-4e61-962d-8178ae088455 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:57.868737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c82f8da-5194-4e61-962d-8178ae088455 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:57.870058 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 743/1479] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:56:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 13:56:57.875335 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137d80b8-81ba-41fe-a865-a03d6c0e5445 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:57.875736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137d80b8-81ba-41fe-a865-a03d6c0e5445 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:57.875908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137d80b8-81ba-41fe-a865-a03d6c0e5445 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:57.941458 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-137d80b8-81ba-41fe-a865-a03d6c0e5445 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:56:57.942607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-137d80b8-81ba-41fe-a865-a03d6c0e5445 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:58.040127 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 737/1480] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:56:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:56:58.333266 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c82f8da-5194-4e61-962d-8178ae088455 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:56:58.334863 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-3c82f8da-5194-4e61-962d-8178ae088455 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:58.344567 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:58.481442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c82f8da-5194-4e61-962d-8178ae088455 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:56:58.560731 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:56:58.597970 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cef950-f734-4403-9d13-4bf16472c7c9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:56:58.598603 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cef950-f734-4403-9d13-4bf16472c7c9 service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:56:58.598882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cef950-f734-4403-9d13-4bf16472c7c9 service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:56:58.599154 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cef950-f734-4403-9d13-4bf16472c7c9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:56:58.600744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-92cef950-f734-4403-9d13-4bf16472c7c9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:56:58.688072 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92cef950-f734-4403-9d13-4bf16472c7c9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:56:58.689648 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92cef950-f734-4403-9d13-4bf16472c7c9 service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:56:58.707975 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 744/1481] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:56:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:57:01.067488 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d5d9e6-099a-4d50-9466-dda69e8f515d None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:57:01.068068 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d5d9e6-099a-4d50-9466-dda69e8f515d None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:57:01.068068 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d5d9e6-099a-4d50-9466-dda69e8f515d None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:57:01.069377 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 738/1482] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:57:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:57:01.074687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7b8844-c21a-452f-a6ab-b0d3fd0e1967 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:57:01.075130 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7b8844-c21a-452f-a6ab-b0d3fd0e1967 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:57:01.075250 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7b8844-c21a-452f-a6ab-b0d3fd0e1967 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:57:01.140673 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-3c7b8844-c21a-452f-a6ab-b0d3fd0e1967 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:57:01.141879 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-3c7b8844-c21a-452f-a6ab-b0d3fd0e1967 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:57:01.188284 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:57:01.238611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 745/1483] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:57:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:57:01.525772 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d5d9e6-099a-4d50-9466-dda69e8f515d None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:57:01.527379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a0d5d9e6-099a-4d50-9466-dda69e8f515d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:57:01.551629 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:57:01.630237 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d5d9e6-099a-4d50-9466-dda69e8f515d None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:57:01.704678 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ad2de09-5db4-40cf-864c-a45f41b590b8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:57:01.705281 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad2de09-5db4-40cf-864c-a45f41b590b8 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:57:01.705557 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad2de09-5db4-40cf-864c-a45f41b590b8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:57:01.706367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad2de09-5db4-40cf-864c-a45f41b590b8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:57:01.708206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9ad2de09-5db4-40cf-864c-a45f41b590b8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:57:01.781574 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:57:01.816347 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad2de09-5db4-40cf-864c-a45f41b590b8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:57:01.817787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ad2de09-5db4-40cf-864c-a45f41b590b8 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:57:01.836232 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 739/1484] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:57:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:57:04.109799 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3299fa4f-f8f6-41c9-bfd9-d151f98104db None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:57:04.110469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3299fa4f-f8f6-41c9-bfd9-d151f98104db None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:57:04.110469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3299fa4f-f8f6-41c9-bfd9-d151f98104db None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:57:04.111825 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 746/1485] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:57:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:57:04.117647 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78bf27e-938c-48c7-bedc-946cb4ee263b None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:57:04.118069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78bf27e-938c-48c7-bedc-946cb4ee263b None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:57:04.118190 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78bf27e-938c-48c7-bedc-946cb4ee263b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:57:04.183363 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-d78bf27e-938c-48c7-bedc-946cb4ee263b None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:57:04.184563 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d78bf27e-938c-48c7-bedc-946cb4ee263b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:57:04.281180 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 740/1486] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:57:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:57:04.568138 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3299fa4f-f8f6-41c9-bfd9-d151f98104db None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:57:04.569657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-3299fa4f-f8f6-41c9-bfd9-d151f98104db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:57:04.672879 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3299fa4f-f8f6-41c9-bfd9-d151f98104db None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:57:04.742908 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:57:04.744434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4801b876-0942-488c-84ce-b06e3006a6d5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:57:04.745027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4801b876-0942-488c-84ce-b06e3006a6d5 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:57:04.745305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4801b876-0942-488c-84ce-b06e3006a6d5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:57:04.745571 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4801b876-0942-488c-84ce-b06e3006a6d5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:57:04.747160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-4801b876-0942-488c-84ce-b06e3006a6d5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:57:04.856206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4801b876-0942-488c-84ce-b06e3006a6d5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:57:04.857535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4801b876-0942-488c-84ce-b06e3006a6d5 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:57:04.875494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 747/1487] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:57:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:57:07.126795 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e10f0aa-6438-4dca-9b3f-ebd85cafd133 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:57:07.128036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e10f0aa-6438-4dca-9b3f-ebd85cafd133 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:57:07.128036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e10f0aa-6438-4dca-9b3f-ebd85cafd133 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:57:07.129506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 741/1488] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:57:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:57:07.135120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960a35ef-2bac-4002-9660-fd6c68c86e35 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:57:07.135532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960a35ef-2bac-4002-9660-fd6c68c86e35 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:57:07.135667 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-960a35ef-2bac-4002-9660-fd6c68c86e35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:57:07.201713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-960a35ef-2bac-4002-9660-fd6c68c86e35 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:57:07.203020 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-960a35ef-2bac-4002-9660-fd6c68c86e35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:57:07.255734 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:57:07.299701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 748/1489] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:57:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:57:07.593348 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e10f0aa-6438-4dca-9b3f-ebd85cafd133 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:57:07.594926 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9e10f0aa-6438-4dca-9b3f-ebd85cafd133 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:57:07.631312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:57:07.699090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e10f0aa-6438-4dca-9b3f-ebd85cafd133 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:57:07.777876 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2eb8297-a9f5-42d4-ac39-444ba602e4ef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:57:07.778508 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2eb8297-a9f5-42d4-ac39-444ba602e4ef service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:57:07.778785 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2eb8297-a9f5-42d4-ac39-444ba602e4ef service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:57:07.779052 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2eb8297-a9f5-42d4-ac39-444ba602e4ef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:57:07.780626 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e2eb8297-a9f5-42d4-ac39-444ba602e4ef service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:57:07.861105 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:57:07.889644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2eb8297-a9f5-42d4-ac39-444ba602e4ef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:57:07.891093 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2eb8297-a9f5-42d4-ac39-444ba602e4ef service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:57:07.910175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 742/1490] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:57:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:57:10.548308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc303d0-5089-47a1-b1e0-9c19e96dcf4f None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:57:10.548875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc303d0-5089-47a1-b1e0-9c19e96dcf4f None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:57:10.548875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc303d0-5089-47a1-b1e0-9c19e96dcf4f None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:57:10.550720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 749/1491] 10.222.0.62 () {54 vars in 927 bytes} [Fri Sep 23 13:57:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:57:10.556011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab69fa5-332d-438e-937a-1ac21cc5d011 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:57:10.556467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab69fa5-332d-438e-937a-1ac21cc5d011 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:57:10.556571 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab69fa5-332d-438e-937a-1ac21cc5d011 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:57:10.621619 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-2ab69fa5-332d-438e-937a-1ac21cc5d011 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:57:10.622852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2ab69fa5-332d-438e-937a-1ac21cc5d011 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:57:10.718713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 743/1492] 10.222.0.62 () {58 vars in 1024 bytes} [Fri Sep 23 13:57:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:57:11.014872 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc303d0-5089-47a1-b1e0-9c19e96dcf4f None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:57:11.016436 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-ddc303d0-5089-47a1-b1e0-9c19e96dcf4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:57:11.117979 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc303d0-5089-47a1-b1e0-9c19e96dcf4f None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:57:11.191996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff05ae4a-c785-4923-929e-65a60e64e26f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:57:11.192672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff05ae4a-c785-4923-929e-65a60e64e26f service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:57:11.193100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff05ae4a-c785-4923-929e-65a60e64e26f service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:57:11.193373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff05ae4a-c785-4923-929e-65a60e64e26f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:57:11.195011 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-ff05ae4a-c785-4923-929e-65a60e64e26f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:57:11.203846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:57:11.305464 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff05ae4a-c785-4923-929e-65a60e64e26f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:57:11.306932 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff05ae4a-c785-4923-929e-65a60e64e26f service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:57:11.324985 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 750/1493] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:57:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:57:20.717087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=82 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:57:21.322707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=50 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:58:32.890397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b876f0f-2038-4f36-89c7-553d4d0f7719 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:58:32.891000 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b876f0f-2038-4f36-89c7-553d4d0f7719 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:58:32.891000 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b876f0f-2038-4f36-89c7-553d4d0f7719 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:58:32.965086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-9b876f0f-2038-4f36-89c7-553d4d0f7719 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:58:32.966932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9b876f0f-2038-4f36-89c7-553d4d0f7719 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:58:33.061609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 744/1494] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 13:58:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:58:33.069757 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428e7722-6d6a-4f22-bd58-d40f892081a7 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:58:33.070235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428e7722-6d6a-4f22-bd58-d40f892081a7 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:58:33.070366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428e7722-6d6a-4f22-bd58-d40f892081a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:58:33.146744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-428e7722-6d6a-4f22-bd58-d40f892081a7 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:58:33.147855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-428e7722-6d6a-4f22-bd58-d40f892081a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:58:33.242726 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 751/1495] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 13:58:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:58:33.247853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b876f0f-2038-4f36-89c7-553d4d0f7719 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:58:33.249111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9b876f0f-2038-4f36-89c7-553d4d0f7719 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:58:33.260434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b876f0f-2038-4f36-89c7-553d4d0f7719 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:58:33.321546 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:58:33.357112 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e5d9bf-2ed1-4935-a066-0fbb3162d315 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:58:33.357709 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e5d9bf-2ed1-4935-a066-0fbb3162d315 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:58:33.357984 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e5d9bf-2ed1-4935-a066-0fbb3162d315 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:58:33.358250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e5d9bf-2ed1-4935-a066-0fbb3162d315 admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:58:33.358937 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e5d9bf-2ed1-4935-a066-0fbb3162d315 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 13:58:33.359237 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e5d9bf-2ed1-4935-a066-0fbb3162d315 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:58:33.362279 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: WARNING py.warnings [None req-f0e5d9bf-2ed1-4935-a066-0fbb3162d315 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 23 13:58:33.362279 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: warnings.warn(msg) Sep 23 13:58:33.362279 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: Sep 23 13:58:33.362279 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e5d9bf-2ed1-4935-a066-0fbb3162d315 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:58:33.373203 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 745/1496] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 13:58:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 13:58:33.381535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428e7722-6d6a-4f22-bd58-d40f892081a7 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:58:33.391143 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428e7722-6d6a-4f22-bd58-d40f892081a7 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:58:33.452539 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:58:33.486151 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de46f57-b94b-4086-8cd2-79a925a2efc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:58:33.486773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de46f57-b94b-4086-8cd2-79a925a2efc5 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:58:33.487048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de46f57-b94b-4086-8cd2-79a925a2efc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:58:33.487314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de46f57-b94b-4086-8cd2-79a925a2efc5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:58:33.487947 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de46f57-b94b-4086-8cd2-79a925a2efc5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:58:33.490202 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: WARNING py.warnings [None req-0de46f57-b94b-4086-8cd2-79a925a2efc5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 23 13:58:33.490202 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: warnings.warn(msg) Sep 23 13:58:33.490202 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: Sep 23 13:58:33.491443 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de46f57-b94b-4086-8cd2-79a925a2efc5 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:58:33.627404 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 752/1497] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 13:58:33 2022] POST /identity/v3/projects => generated 379 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:58:33.631804 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e5d9bf-2ed1-4935-a066-0fbb3162d315 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:58:33.633346 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f0e5d9bf-2ed1-4935-a066-0fbb3162d315 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:58:33.666533 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e5d9bf-2ed1-4935-a066-0fbb3162d315 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:58:33.761066 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3a7955-a897-4761-a3f8-a6b7a51960b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:58:33.761713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3a7955-a897-4761-a3f8-a6b7a51960b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:58:33.762015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3a7955-a897-4761-a3f8-a6b7a51960b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:58:33.762284 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3a7955-a897-4761-a3f8-a6b7a51960b2 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:58:33.762956 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3a7955-a897-4761-a3f8-a6b7a51960b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:58:33.763946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: WARNING py.warnings [None req-6c3a7955-a897-4761-a3f8-a6b7a51960b2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 23 13:58:33.763946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: warnings.warn(msg) Sep 23 13:58:33.763946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: Sep 23 13:58:33.765144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3a7955-a897-4761-a3f8-a6b7a51960b2 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:58:33.855168 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 746/1498] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 13:58:33 2022] POST /identity/v3/users => generated 290 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 13:58:33.860063 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de46f57-b94b-4086-8cd2-79a925a2efc5 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:58:33.861560 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-0de46f57-b94b-4086-8cd2-79a925a2efc5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:58:33.922034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de46f57-b94b-4086-8cd2-79a925a2efc5 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:58:34.005996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:58:34.015853 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1663d6-18b3-4db4-ba1a-8634e366ed6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:58:34.024479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1663d6-18b3-4db4-ba1a-8634e366ed6e admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:58:34.024479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1663d6-18b3-4db4-ba1a-8634e366ed6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:58:34.024479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1663d6-18b3-4db4-ba1a-8634e366ed6e admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:58:34.024479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1663d6-18b3-4db4-ba1a-8634e366ed6e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:58:34.024479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: WARNING py.warnings [None req-1f1663d6-18b3-4db4-ba1a-8634e366ed6e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 23 13:58:34.024479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: warnings.warn(msg) Sep 23 13:58:34.024479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: Sep 23 13:58:34.025085 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1663d6-18b3-4db4-ba1a-8634e366ed6e admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:58:34.033853 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 753/1499] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 13:58:33 2022] GET /identity/v3/roles => generated 1420 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 13:58:34.037924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3a7955-a897-4761-a3f8-a6b7a51960b2 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:58:34.039459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-6c3a7955-a897-4761-a3f8-a6b7a51960b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:58:34.051424 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3a7955-a897-4761-a3f8-a6b7a51960b2 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:58:34.076186 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:58:34.147467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9df084c-6a9d-41e1-8173-bf27baedaf3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:58:34.148069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9df084c-6a9d-41e1-8173-bf27baedaf3d admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:58:34.148341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9df084c-6a9d-41e1-8173-bf27baedaf3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:58:34.148607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9df084c-6a9d-41e1-8173-bf27baedaf3d admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:58:34.149244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9df084c-6a9d-41e1-8173-bf27baedaf3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:58:34.150241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: WARNING py.warnings [None req-c9df084c-6a9d-41e1-8173-bf27baedaf3d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 23 13:58:34.150241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: warnings.warn(msg) Sep 23 13:58:34.150241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: Sep 23 13:58:34.151147 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9df084c-6a9d-41e1-8173-bf27baedaf3d admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:58:34.165687 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 747/1500] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 13:58:34 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 13:58:34.170992 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1663d6-18b3-4db4-ba1a-8634e366ed6e admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:58:34.172502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-1f1663d6-18b3-4db4-ba1a-8634e366ed6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:58:34.208003 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1663d6-18b3-4db4-ba1a-8634e366ed6e admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:58:34.301526 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7ee7fc-96a7-4e04-b468-cbb5edcf637c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:58:34.302104 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7ee7fc-96a7-4e04-b468-cbb5edcf637c admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:58:34.302418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7ee7fc-96a7-4e04-b468-cbb5edcf637c admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:58:34.302691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7ee7fc-96a7-4e04-b468-cbb5edcf637c admin admin] PATH_INFO: `/v3/projects/60d2e4d0d650436b846bcc4f0c627e61/users/ef83b765f64a4e4ba1f0830680665d9e/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:58:34.324387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7ee7fc-96a7-4e04-b468-cbb5edcf637c admin admin] RBAC: Authorizing `identity:create_grant(project_id=60d2e4d0d650436b846bcc4f0c627e61, user_id=ef83b765f64a4e4ba1f0830680665d9e, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:58:34.325684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: WARNING py.warnings [None req-5b7ee7fc-96a7-4e04-b468-cbb5edcf637c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 23 13:58:34.325684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: warnings.warn(msg) Sep 23 13:58:34.325684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: Sep 23 13:58:34.327660 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7ee7fc-96a7-4e04-b468-cbb5edcf637c admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:58:34.402998 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 754/1501] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 13:58:34 2022] PUT /identity/v3/projects/60d2e4d0d650436b846bcc4f0c627e61/users/ef83b765f64a4e4ba1f0830680665d9e/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 13:58:34.415832 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4e5878-f04b-4473-a442-3e64085c5896 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:58:34.416117 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4e5878-f04b-4473-a442-3e64085c5896 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:58:34.416389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed4e5878-f04b-4473-a442-3e64085c5896 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:58:34.472988 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-ed4e5878-f04b-4473-a442-3e64085c5896 None None] MFA Rules not processed for user `ef83b765f64a4e4ba1f0830680665d9e`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:58:34.474191 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ed4e5878-f04b-4473-a442-3e64085c5896 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:58:34.550985 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 748/1502] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 13:58:34 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:58:34.564200 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893761ce-baef-4727-b080-c4f44e86d362 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:58:34.564483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893761ce-baef-4727-b080-c4f44e86d362 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:58:34.564753 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893761ce-baef-4727-b080-c4f44e86d362 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:58:34.566086 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 755/1503] 10.222.0.62 () {54 vars in 963 bytes} [Fri Sep 23 13:58:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:58:34.570842 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051ae22f-2dda-4976-9548-54159be27653 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:58:34.571125 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051ae22f-2dda-4976-9548-54159be27653 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:58:34.571395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051ae22f-2dda-4976-9548-54159be27653 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:58:34.603301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:58:34.647674 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-051ae22f-2dda-4976-9548-54159be27653 None None] MFA Rules not processed for user `fae68ca0754a4b0b88a57f39efa2fb45`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:58:34.648863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-051ae22f-2dda-4976-9548-54159be27653 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:58:34.723209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 749/1504] 10.222.0.62 () {58 vars in 1060 bytes} [Fri Sep 23 13:58:34 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 13:58:34.730977 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893761ce-baef-4727-b080-c4f44e86d362 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:58:34.732617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-893761ce-baef-4727-b080-c4f44e86d362 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:58:34.782102 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:58:34.834694 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893761ce-baef-4727-b080-c4f44e86d362 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:58:34.907381 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e7429d-563e-4004-aa30-4006574bc603 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:58:34.907976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e7429d-563e-4004-aa30-4006574bc603 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:58:34.908253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e7429d-563e-4004-aa30-4006574bc603 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:58:34.908520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e7429d-563e-4004-aa30-4006574bc603 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:58:34.910090 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c9e7429d-563e-4004-aa30-4006574bc603 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:58:35.008980 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e7429d-563e-4004-aa30-4006574bc603 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:58:35.010429 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e7429d-563e-4004-aa30-4006574bc603 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:58:35.014576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:58:35.028466 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 756/1505] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:58:34 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:58:35.073967 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932f85bf-2b3a-4ac0-ab3e-428c2732aadc None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:58:35.074683 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932f85bf-2b3a-4ac0-ab3e-428c2732aadc None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:58:35.074829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-932f85bf-2b3a-4ac0-ab3e-428c2732aadc None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:58:35.076782 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 750/1506] 10.222.0.62 () {52 vars in 830 bytes} [Fri Sep 23 13:58:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 13:58:35.080456 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e7429d-563e-4004-aa30-4006574bc603 service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:58:35.081667 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c9e7429d-563e-4004-aa30-4006574bc603 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:58:35.178122 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e7429d-563e-4004-aa30-4006574bc603 service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:58:35.270875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407c90a6-76b3-4f44-9dcc-aa8abf0e1fbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:58:35.271494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407c90a6-76b3-4f44-9dcc-aa8abf0e1fbf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:58:35.271769 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407c90a6-76b3-4f44-9dcc-aa8abf0e1fbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:58:35.272037 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407c90a6-76b3-4f44-9dcc-aa8abf0e1fbf admin admin] PATH_INFO: `/v3/users/ef83b765f64a4e4ba1f0830680665d9e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:58:35.284694 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407c90a6-76b3-4f44-9dcc-aa8abf0e1fbf admin admin] RBAC: Authorizing `identity:delete_user(user_id=ef83b765f64a4e4ba1f0830680665d9e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:58:35.285707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: WARNING py.warnings [None req-407c90a6-76b3-4f44-9dcc-aa8abf0e1fbf admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 23 13:58:35.285707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: warnings.warn(msg) Sep 23 13:58:35.285707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: Sep 23 13:58:35.286946 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407c90a6-76b3-4f44-9dcc-aa8abf0e1fbf admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:58:35.297219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:58:35.531671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-407c90a6-76b3-4f44-9dcc-aa8abf0e1fbf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ef83b765f64a4e4ba1f0830680665d9e'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 13:58:35.608157 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-407c90a6-76b3-4f44-9dcc-aa8abf0e1fbf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ef83b765f64a4e4ba1f0830680665d9e'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 13:58:35.631003 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-407c90a6-76b3-4f44-9dcc-aa8abf0e1fbf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ef83b765f64a4e4ba1f0830680665d9e'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 13:58:35.634849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-407c90a6-76b3-4f44-9dcc-aa8abf0e1fbf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ef83b765f64a4e4ba1f0830680665d9e'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 13:58:35.646066 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-407c90a6-76b3-4f44-9dcc-aa8abf0e1fbf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ef83b765f64a4e4ba1f0830680665d9e'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 13:58:35.652485 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 757/1507] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 13:58:35 2022] DELETE /identity/v3/users/ef83b765f64a4e4ba1f0830680665d9e => generated 0 bytes in 572 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:58:35.657764 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3701aac0-793c-4944-985c-66b8ca091cf1 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:58:35.658050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3701aac0-793c-4944-985c-66b8ca091cf1 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:58:35.658321 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3701aac0-793c-4944-985c-66b8ca091cf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:58:35.713941 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-3701aac0-793c-4944-985c-66b8ca091cf1 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 13:58:35.715153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3701aac0-793c-4944-985c-66b8ca091cf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:58:35.790085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:58:35.802379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 751/1508] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 13:58:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 13:58:35.812175 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407c90a6-76b3-4f44-9dcc-aa8abf0e1fbf admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:58:35.813362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-407c90a6-76b3-4f44-9dcc-aa8abf0e1fbf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:58:35.913966 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407c90a6-76b3-4f44-9dcc-aa8abf0e1fbf admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:58:35.967453 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:58:35.986059 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2c23b36-8b08-4582-8b3f-0692ce70d6c9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:58:35.986689 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c23b36-8b08-4582-8b3f-0692ce70d6c9 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:58:35.986962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c23b36-8b08-4582-8b3f-0692ce70d6c9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:58:35.987226 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2c23b36-8b08-4582-8b3f-0692ce70d6c9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:58:35.988789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d2c23b36-8b08-4582-8b3f-0692ce70d6c9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:58:36.101068 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c23b36-8b08-4582-8b3f-0692ce70d6c9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:58:36.102444 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2c23b36-8b08-4582-8b3f-0692ce70d6c9 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:58:36.120765 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 758/1509] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 13:58:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 13:58:36.581673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3701aac0-793c-4944-985c-66b8ca091cf1 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 13:58:36.583059 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3701aac0-793c-4944-985c-66b8ca091cf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 13:58:36.595461 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3701aac0-793c-4944-985c-66b8ca091cf1 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 13:58:36.691988 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724020d0-b0a2-4649-a7ea-8f3034bb773d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 13:58:36.692614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724020d0-b0a2-4649-a7ea-8f3034bb773d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 13:58:36.692887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724020d0-b0a2-4649-a7ea-8f3034bb773d admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 13:58:36.693148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724020d0-b0a2-4649-a7ea-8f3034bb773d admin admin] PATH_INFO: `/v3/projects/60d2e4d0d650436b846bcc4f0c627e61` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 13:58:36.703926 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724020d0-b0a2-4649-a7ea-8f3034bb773d admin admin] RBAC: Authorizing `identity:delete_project(project_id=60d2e4d0d650436b846bcc4f0c627e61)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 13:58:36.704919 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: WARNING py.warnings [None req-724020d0-b0a2-4649-a7ea-8f3034bb773d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 23 13:58:36.704919 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: warnings.warn(msg) Sep 23 13:58:36.704919 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: Sep 23 13:58:36.706151 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724020d0-b0a2-4649-a7ea-8f3034bb773d admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 13:58:36.808685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-724020d0-b0a2-4649-a7ea-8f3034bb773d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '60d2e4d0d650436b846bcc4f0c627e61', 'request_id': 'req-724020d0-b0a2-4649-a7ea-8f3034bb773d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 13:58:36.858095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-724020d0-b0a2-4649-a7ea-8f3034bb773d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '60d2e4d0d650436b846bcc4f0c627e61', 'request_id': 'req-724020d0-b0a2-4649-a7ea-8f3034bb773d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 13:58:36.864290 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-724020d0-b0a2-4649-a7ea-8f3034bb773d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '60d2e4d0d650436b846bcc4f0c627e61', 'request_id': 'req-724020d0-b0a2-4649-a7ea-8f3034bb773d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 13:58:36.895051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-724020d0-b0a2-4649-a7ea-8f3034bb773d admin admin] The token cache is being invalidate because project 60d2e4d0d650436b846bcc4f0c627e61 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 13:58:36.895268 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-724020d0-b0a2-4649-a7ea-8f3034bb773d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 13:58:36.895690 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-724020d0-b0a2-4649-a7ea-8f3034bb773d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 13:58:36.897070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 752/1510] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 13:58:36 2022] DELETE /identity/v3/projects/60d2e4d0d650436b846bcc4f0c627e61 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 13:58:46.118575 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=62 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 13:58:46.893001 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:00:07.778439 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281a855e-79c9-4b90-af67-c173fb022923 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:07.778889 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281a855e-79c9-4b90-af67-c173fb022923 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:07.779113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281a855e-79c9-4b90-af67-c173fb022923 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:07.780386 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 759/1511] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 23 14:00:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:00:07.785183 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd85e9f2-f38c-45ec-841c-0ea66d874335 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:07.785582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd85e9f2-f38c-45ec-841c-0ea66d874335 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:07.785729 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd85e9f2-f38c-45ec-841c-0ea66d874335 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:07.866019 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-bd85e9f2-f38c-45ec-841c-0ea66d874335 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:00:07.867258 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-bd85e9f2-f38c-45ec-841c-0ea66d874335 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:00:07.969656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 753/1512] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 23 14:00:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:00:08.123706 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281a855e-79c9-4b90-af67-c173fb022923 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:00:08.130991 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-281a855e-79c9-4b90-af67-c173fb022923 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:00:08.261171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281a855e-79c9-4b90-af67-c173fb022923 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:00:08.266790 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e848c32d-6893-4a1c-b863-32098ef18bb8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:00:08.267400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e848c32d-6893-4a1c-b863-32098ef18bb8 service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:08.267685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e848c32d-6893-4a1c-b863-32098ef18bb8 service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:08.267986 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e848c32d-6893-4a1c-b863-32098ef18bb8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:08.276923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e848c32d-6893-4a1c-b863-32098ef18bb8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:00:08.278548 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e848c32d-6893-4a1c-b863-32098ef18bb8 service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:00:08.288483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 760/1513] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:00:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:00:08.927539 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4e1471-05ce-43e5-b87e-e1903c1fc216 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:08.928201 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4e1471-05ce-43e5-b87e-e1903c1fc216 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:08.928201 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4e1471-05ce-43e5-b87e-e1903c1fc216 None None] PATH_INFO: `/v3` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:08.929354 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 754/1514] 10.222.0.49 () {54 vars in 925 bytes} [Fri Sep 23 14:00:08 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:00:08.935858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e778e8-b343-4e5f-a61c-971f040a4339 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:08.936292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e778e8-b343-4e5f-a61c-971f040a4339 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:08.936384 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e778e8-b343-4e5f-a61c-971f040a4339 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:08.999947 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-22e778e8-b343-4e5f-a61c-971f040a4339 None None] MFA Rules not processed for user `ce249d4a99c34891b6ba941f57a14dd4`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:00:09.001145 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-22e778e8-b343-4e5f-a61c-971f040a4339 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:00:09.069377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:00:09.075101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 761/1515] 10.222.0.49 () {58 vars in 1015 bytes} [Fri Sep 23 14:00:08 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:00:09.376893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030c0e83-e664-4475-8190-736af3924977 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:09.377178 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030c0e83-e664-4475-8190-736af3924977 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:09.377455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030c0e83-e664-4475-8190-736af3924977 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:09.378924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 755/1516] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 23 14:00:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:00:09.385901 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f9a04a-8c92-4656-97da-0e66c44040ba None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:09.386184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f9a04a-8c92-4656-97da-0e66c44040ba None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:09.386493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f9a04a-8c92-4656-97da-0e66c44040ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:09.405714 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-b7f9a04a-8c92-4656-97da-0e66c44040ba None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:00:09.406932 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-b7f9a04a-8c92-4656-97da-0e66c44040ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:00:09.414802 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 762/1517] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 23 14:00:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:00:09.568498 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030c0e83-e664-4475-8190-736af3924977 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:00:09.582581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030c0e83-e664-4475-8190-736af3924977 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:00:09.588007 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f0c475-07fb-490a-b1e0-df316eec6673 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:00:09.588762 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f0c475-07fb-490a-b1e0-df316eec6673 service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:09.589045 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f0c475-07fb-490a-b1e0-df316eec6673 service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:09.589319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f0c475-07fb-490a-b1e0-df316eec6673 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:09.597148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f0c475-07fb-490a-b1e0-df316eec6673 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:00:09.598738 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f0c475-07fb-490a-b1e0-df316eec6673 service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:00:09.608564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 756/1518] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:00:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:00:18.110456 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d025deb6-9d79-46f2-b375-6913df045ace None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:18.111024 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d025deb6-9d79-46f2-b375-6913df045ace None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:18.111024 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d025deb6-9d79-46f2-b375-6913df045ace None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:18.112350 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 763/1519] 10.222.0.62 () {54 vars in 961 bytes} [Fri Sep 23 14:00:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:00:18.117334 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975ea95d-9e6d-496e-83cf-f3ea351e4494 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:18.117729 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975ea95d-9e6d-496e-83cf-f3ea351e4494 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:18.117929 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975ea95d-9e6d-496e-83cf-f3ea351e4494 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:18.138346 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-975ea95d-9e6d-496e-83cf-f3ea351e4494 None None] MFA Rules not processed for user `ce249d4a99c34891b6ba941f57a14dd4`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:00:18.139576 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-975ea95d-9e6d-496e-83cf-f3ea351e4494 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:00:18.148380 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 757/1520] 10.222.0.62 () {58 vars in 1058 bytes} [Fri Sep 23 14:00:18 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:00:18.156678 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d025deb6-9d79-46f2-b375-6913df045ace None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:00:18.170441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d025deb6-9d79-46f2-b375-6913df045ace None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:00:18.175329 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46cecd1-dec8-4023-b746-d6f9ca5158c1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce249d4a99c34891b6ba941f57a14dd4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:00:18.175911 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46cecd1-dec8-4023-b746-d6f9ca5158c1 service placement] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:18.176188 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46cecd1-dec8-4023-b746-d6f9ca5158c1 service placement] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:18.176456 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46cecd1-dec8-4023-b746-d6f9ca5158c1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:18.184547 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46cecd1-dec8-4023-b746-d6f9ca5158c1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:00:18.186100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46cecd1-dec8-4023-b746-d6f9ca5158c1 service placement] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:00:18.195825 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 764/1521] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:00:18 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:00:21.119665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0662b5-7243-4870-9b03-f5bf6f936ab7 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:21.120285 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0662b5-7243-4870-9b03-f5bf6f936ab7 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:21.120285 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0662b5-7243-4870-9b03-f5bf6f936ab7 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:21.122078 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 758/1522] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 23 14:00:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:00:21.128146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c54863b-6fa0-4044-be53-8b26fc909e2b None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:21.128589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c54863b-6fa0-4044-be53-8b26fc909e2b None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:21.128708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c54863b-6fa0-4044-be53-8b26fc909e2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:21.149177 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-0c54863b-6fa0-4044-be53-8b26fc909e2b None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:00:21.150412 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-0c54863b-6fa0-4044-be53-8b26fc909e2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:00:21.158148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 765/1523] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 23 14:00:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:00:21.315210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0662b5-7243-4870-9b03-f5bf6f936ab7 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:00:21.330422 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0662b5-7243-4870-9b03-f5bf6f936ab7 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:00:21.335860 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676b63c1-f43d-46db-919e-a48282ee3c98 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:00:21.336450 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676b63c1-f43d-46db-919e-a48282ee3c98 service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:21.336726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676b63c1-f43d-46db-919e-a48282ee3c98 service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:21.336997 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676b63c1-f43d-46db-919e-a48282ee3c98 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:21.345164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676b63c1-f43d-46db-919e-a48282ee3c98 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:00:21.346773 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676b63c1-f43d-46db-919e-a48282ee3c98 service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:00:21.356141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 759/1524] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:00:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:00:22.437188 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e17a84d-11c7-43e5-ad73-03c2497bdcf7 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:22.437760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e17a84d-11c7-43e5-ad73-03c2497bdcf7 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:22.437760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e17a84d-11c7-43e5-ad73-03c2497bdcf7 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:22.439088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 766/1525] 127.0.0.1 () {54 vars in 901 bytes} [Fri Sep 23 14:00:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:00:22.445356 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33385ab-fb5a-4f4c-9f8e-bec4410ca7ae None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:22.445757 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33385ab-fb5a-4f4c-9f8e-bec4410ca7ae None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:22.445904 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33385ab-fb5a-4f4c-9f8e-bec4410ca7ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:22.466485 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-a33385ab-fb5a-4f4c-9f8e-bec4410ca7ae None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:00:22.467644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a33385ab-fb5a-4f4c-9f8e-bec4410ca7ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:00:22.475493 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 760/1526] 127.0.0.1 () {58 vars in 998 bytes} [Fri Sep 23 14:00:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:00:22.628353 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e17a84d-11c7-43e5-ad73-03c2497bdcf7 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:00:22.643215 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e17a84d-11c7-43e5-ad73-03c2497bdcf7 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:00:22.648578 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6f6ed4-6b3d-4b7a-bcfa-8465276fe466 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:00:22.649160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6f6ed4-6b3d-4b7a-bcfa-8465276fe466 service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:22.649438 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6f6ed4-6b3d-4b7a-bcfa-8465276fe466 service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:22.649709 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6f6ed4-6b3d-4b7a-bcfa-8465276fe466 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:22.658065 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6f6ed4-6b3d-4b7a-bcfa-8465276fe466 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:00:22.659693 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6f6ed4-6b3d-4b7a-bcfa-8465276fe466 service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:00:22.668968 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 767/1527] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:00:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:00:26.655521 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e62a489-9a02-430c-ba17-ae12dd8384b6 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:26.656067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e62a489-9a02-430c-ba17-ae12dd8384b6 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:26.656067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e62a489-9a02-430c-ba17-ae12dd8384b6 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:26.657438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 761/1528] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 23 14:00:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:00:26.662615 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64587121-d8ad-456c-989d-7a03832dc1de None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:26.663060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64587121-d8ad-456c-989d-7a03832dc1de None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:26.663158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64587121-d8ad-456c-989d-7a03832dc1de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:26.683342 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-64587121-d8ad-456c-989d-7a03832dc1de None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:00:26.684519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-64587121-d8ad-456c-989d-7a03832dc1de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:00:26.692403 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 768/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 23 14:00:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:00:26.979909 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e62a489-9a02-430c-ba17-ae12dd8384b6 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:00:26.981338 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-7e62a489-9a02-430c-ba17-ae12dd8384b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:00:27.013977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:00:27.061959 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e62a489-9a02-430c-ba17-ae12dd8384b6 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:00:27.065805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68be96f-b836-43a7-b209-e57c17515aa3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:00:27.066437 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68be96f-b836-43a7-b209-e57c17515aa3 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:27.066722 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68be96f-b836-43a7-b209-e57c17515aa3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:27.066996 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68be96f-b836-43a7-b209-e57c17515aa3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:27.077170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68be96f-b836-43a7-b209-e57c17515aa3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:00:27.078626 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68be96f-b836-43a7-b209-e57c17515aa3 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:00:27.088307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 762/1530] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:00:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:00:29.239098 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb666159-d6f9-4e74-9384-fac059da18ec None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:29.239551 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb666159-d6f9-4e74-9384-fac059da18ec None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:29.239551 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb666159-d6f9-4e74-9384-fac059da18ec None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:29.240921 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 769/1531] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 23 14:00:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 14:00:29.246230 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f61345-bc4f-4427-9ce9-037795a2f456 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:29.246692 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f61345-bc4f-4427-9ce9-037795a2f456 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:29.246819 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f61345-bc4f-4427-9ce9-037795a2f456 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:29.266840 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-83f61345-bc4f-4427-9ce9-037795a2f456 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:00:29.268040 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-83f61345-bc4f-4427-9ce9-037795a2f456 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:00:29.275871 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 763/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 23 14:00:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:00:29.568083 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb666159-d6f9-4e74-9384-fac059da18ec None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:00:29.569789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-fb666159-d6f9-4e74-9384-fac059da18ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:00:29.582880 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb666159-d6f9-4e74-9384-fac059da18ec None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:00:29.586491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d1344b-901a-44db-9728-b784ce3e7319 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:00:29.587082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d1344b-901a-44db-9728-b784ce3e7319 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:29.587388 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d1344b-901a-44db-9728-b784ce3e7319 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:29.587656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d1344b-901a-44db-9728-b784ce3e7319 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:29.597016 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d1344b-901a-44db-9728-b784ce3e7319 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:00:29.598453 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d1344b-901a-44db-9728-b784ce3e7319 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:00:29.608302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 770/1533] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:00:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:00:33.278854 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37edc8c8-fd80-4f71-bcef-263644f30846 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:33.279371 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37edc8c8-fd80-4f71-bcef-263644f30846 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:33.279371 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37edc8c8-fd80-4f71-bcef-263644f30846 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:33.280747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 764/1534] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 23 14:00:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:00:33.285976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c3d5b4-32b8-486d-add8-13cae54854fd None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:33.286467 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c3d5b4-32b8-486d-add8-13cae54854fd None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:33.286564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c3d5b4-32b8-486d-add8-13cae54854fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:33.305758 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-52c3d5b4-32b8-486d-add8-13cae54854fd None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:00:33.307483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-52c3d5b4-32b8-486d-add8-13cae54854fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:00:33.315616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 771/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 23 14:00:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:00:33.628130 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37edc8c8-fd80-4f71-bcef-263644f30846 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:00:33.629851 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-37edc8c8-fd80-4f71-bcef-263644f30846 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:00:33.708633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37edc8c8-fd80-4f71-bcef-263644f30846 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:00:33.712245 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa761aa-e3b7-41c0-b284-bd4b2818ab4b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:00:33.712840 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa761aa-e3b7-41c0-b284-bd4b2818ab4b service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:33.713123 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa761aa-e3b7-41c0-b284-bd4b2818ab4b service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:33.713392 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa761aa-e3b7-41c0-b284-bd4b2818ab4b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:33.722120 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa761aa-e3b7-41c0-b284-bd4b2818ab4b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:00:33.723568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa761aa-e3b7-41c0-b284-bd4b2818ab4b service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:00:33.733262 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 765/1536] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:00:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:00:35.907610 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ea5e81-560e-49ed-bae9-8d4ae1ba437e None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:35.908172 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ea5e81-560e-49ed-bae9-8d4ae1ba437e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:35.908172 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ea5e81-560e-49ed-bae9-8d4ae1ba437e None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:35.909523 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 772/1537] 127.0.0.1 () {54 vars in 917 bytes} [Fri Sep 23 14:00:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:00:35.914838 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3cce5c-d275-4d91-8f63-08331e236e40 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:35.915218 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3cce5c-d275-4d91-8f63-08331e236e40 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:35.915382 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3cce5c-d275-4d91-8f63-08331e236e40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:35.934637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-cb3cce5c-d275-4d91-8f63-08331e236e40 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:00:35.935815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-cb3cce5c-d275-4d91-8f63-08331e236e40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:00:35.943726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 766/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Sep 23 14:00:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:00:36.254992 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ea5e81-560e-49ed-bae9-8d4ae1ba437e None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:00:36.256679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-50ea5e81-560e-49ed-bae9-8d4ae1ba437e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:00:36.270901 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ea5e81-560e-49ed-bae9-8d4ae1ba437e None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:00:36.274504 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ca009f-2f4d-461b-93e2-d35e4e7bdb21 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:00:36.275093 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ca009f-2f4d-461b-93e2-d35e4e7bdb21 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:00:36.275399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ca009f-2f4d-461b-93e2-d35e4e7bdb21 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:00:36.276081 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ca009f-2f4d-461b-93e2-d35e4e7bdb21 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:00:36.284981 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ca009f-2f4d-461b-93e2-d35e4e7bdb21 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:00:36.286457 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ca009f-2f4d-461b-93e2-d35e4e7bdb21 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:00:36.296185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 773/1539] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:00:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:00:45.929549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=63 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:00:46.292026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=37 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:02.355129 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420491b0-eab2-4591-b32b-27aca168ad3a None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:02.355782 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420491b0-eab2-4591-b32b-27aca168ad3a None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:02.355951 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420491b0-eab2-4591-b32b-27aca168ad3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:02.356829 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e628b785-ff4d-4db7-a052-0f3479587dd6 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:02.357268 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e628b785-ff4d-4db7-a052-0f3479587dd6 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:02.357385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e628b785-ff4d-4db7-a052-0f3479587dd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:02.379018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-e628b785-ff4d-4db7-a052-0f3479587dd6 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:02.380615 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-e628b785-ff4d-4db7-a052-0f3479587dd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:02.388258 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-420491b0-eab2-4591-b32b-27aca168ad3a None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:02.389562 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-420491b0-eab2-4591-b32b-27aca168ad3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:02.394431 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 774/1540] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:02.398571 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5df37ef-c0c0-40ea-84f5-ef6a16341f92 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:02.399859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5df37ef-c0c0-40ea-84f5-ef6a16341f92 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:02.400456 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5df37ef-c0c0-40ea-84f5-ef6a16341f92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:02.406359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 767/1541] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:02.412336 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75185119-bffd-4bc9-959d-47dcec00140d None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:02.412618 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75185119-bffd-4bc9-959d-47dcec00140d None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:02.412885 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75185119-bffd-4bc9-959d-47dcec00140d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:02.429254 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-f5df37ef-c0c0-40ea-84f5-ef6a16341f92 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:02.435001 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f5df37ef-c0c0-40ea-84f5-ef6a16341f92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:02.447041 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 775/1542] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:02:02.449827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5df37ef-c0c0-40ea-84f5-ef6a16341f92 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:02.451053 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-75185119-bffd-4bc9-959d-47dcec00140d None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:02.452241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-75185119-bffd-4bc9-959d-47dcec00140d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:02.464332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5df37ef-c0c0-40ea-84f5-ef6a16341f92 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:02.467484 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 768/1543] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Sep 23 14:02:02.470643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b664f924-9c1b-4d4f-9469-de1d6a03e5dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:02.471601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b664f924-9c1b-4d4f-9469-de1d6a03e5dd admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:02.472508 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b664f924-9c1b-4d4f-9469-de1d6a03e5dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:02.472831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b664f924-9c1b-4d4f-9469-de1d6a03e5dd admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:02.473754 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b664f924-9c1b-4d4f-9469-de1d6a03e5dd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:02.474282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b664f924-9c1b-4d4f-9469-de1d6a03e5dd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:02.477730 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75185119-bffd-4bc9-959d-47dcec00140d None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:02.479066 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: WARNING py.warnings [None req-b664f924-9c1b-4d4f-9469-de1d6a03e5dd admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 23 14:02:02.479066 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: warnings.warn(msg) Sep 23 14:02:02.479066 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: Sep 23 14:02:02.480376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b664f924-9c1b-4d4f-9469-de1d6a03e5dd admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:02.489955 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75185119-bffd-4bc9-959d-47dcec00140d None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:02.499002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 776/1544] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 14:02:02.504138 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc37a6f-166e-49b1-ba6b-4148a34c515d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:02.504769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc37a6f-166e-49b1-ba6b-4148a34c515d admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:02.505045 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc37a6f-166e-49b1-ba6b-4148a34c515d admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:02.505310 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc37a6f-166e-49b1-ba6b-4148a34c515d admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:02.505962 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc37a6f-166e-49b1-ba6b-4148a34c515d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:02.506258 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc37a6f-166e-49b1-ba6b-4148a34c515d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:02.508021 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc37a6f-166e-49b1-ba6b-4148a34c515d admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:02.509071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b664f924-9c1b-4d4f-9469-de1d6a03e5dd admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:02.523869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b664f924-9c1b-4d4f-9469-de1d6a03e5dd admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:02.524804 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 769/1545] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 14:02:02.530691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40090b20-5a26-439e-b0b0-3bb0d9671de0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:02.532591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40090b20-5a26-439e-b0b0-3bb0d9671de0 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:02.532823 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc37a6f-166e-49b1-ba6b-4148a34c515d admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:02.533413 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40090b20-5a26-439e-b0b0-3bb0d9671de0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:02.533902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40090b20-5a26-439e-b0b0-3bb0d9671de0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:02.534789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40090b20-5a26-439e-b0b0-3bb0d9671de0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:02.536970 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40090b20-5a26-439e-b0b0-3bb0d9671de0 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:02.550355 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc37a6f-166e-49b1-ba6b-4148a34c515d admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:02.567774 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a481f3-fcef-4404-9259-26c333a52950 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:02.568429 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a481f3-fcef-4404-9259-26c333a52950 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:02.568699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a481f3-fcef-4404-9259-26c333a52950 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:02.568969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a481f3-fcef-4404-9259-26c333a52950 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:02.569605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a481f3-fcef-4404-9259-26c333a52950 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:02.571453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: WARNING py.warnings [None req-c4a481f3-fcef-4404-9259-26c333a52950 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 23 14:02:02.571453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: warnings.warn(msg) Sep 23 14:02:02.571453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: Sep 23 14:02:02.572758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a481f3-fcef-4404-9259-26c333a52950 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:02.682724 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 777/1546] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:02 2022] POST /identity/v3/projects => generated 377 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:02:02.684051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e880a97-8809-420a-b766-419145d83d62 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:02.684338 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e880a97-8809-420a-b766-419145d83d62 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:02.684611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e880a97-8809-420a-b766-419145d83d62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:02.704292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-7e880a97-8809-420a-b766-419145d83d62 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:02.707481 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-7e880a97-8809-420a-b766-419145d83d62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:02.708811 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 770/1547] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:02 2022] POST /identity/v3/projects => generated 393 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:02:02.710719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09774cce-875d-4a6e-bf19-0b2fab254585 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:02.710719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09774cce-875d-4a6e-bf19-0b2fab254585 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:02.710719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09774cce-875d-4a6e-bf19-0b2fab254585 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:02.730927 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-09774cce-875d-4a6e-bf19-0b2fab254585 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:02.732042 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-09774cce-875d-4a6e-bf19-0b2fab254585 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:02.765712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 778/1548] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:02.767026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4265b0de-6b5d-4772-b10e-a42fcf4d7898 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:02.767337 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4265b0de-6b5d-4772-b10e-a42fcf4d7898 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:02.767609 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4265b0de-6b5d-4772-b10e-a42fcf4d7898 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:02.787998 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-4265b0de-6b5d-4772-b10e-a42fcf4d7898 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:02.789203 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-4265b0de-6b5d-4772-b10e-a42fcf4d7898 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:02.797281 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 771/1549] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:02.799263 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debe6a6c-b26c-49e8-a368-e29fc499c06a None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:02.799793 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debe6a6c-b26c-49e8-a368-e29fc499c06a None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:02.800249 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debe6a6c-b26c-49e8-a368-e29fc499c06a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:02.821117 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-debe6a6c-b26c-49e8-a368-e29fc499c06a None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:02.822392 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-debe6a6c-b26c-49e8-a368-e29fc499c06a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:02.830214 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 772/1550] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:02.831484 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc66f9d9-5dd5-4588-a1ea-8164055dff91 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:02.831766 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc66f9d9-5dd5-4588-a1ea-8164055dff91 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:02.832045 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc66f9d9-5dd5-4588-a1ea-8164055dff91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:02.851936 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-bc66f9d9-5dd5-4588-a1ea-8164055dff91 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:02.852641 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 779/1551] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:02.853131 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-bc66f9d9-5dd5-4588-a1ea-8164055dff91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:02.853953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa77a12-5676-4c4a-817d-8932d5a57816 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:02.854237 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa77a12-5676-4c4a-817d-8932d5a57816 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:02.854534 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa77a12-5676-4c4a-817d-8932d5a57816 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:02.868145 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 773/1552] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:02.869288 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc66f9d9-5dd5-4588-a1ea-8164055dff91 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:02.874908 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-faa77a12-5676-4c4a-817d-8932d5a57816 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:02.879162 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-faa77a12-5676-4c4a-817d-8932d5a57816 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:02.880171 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc66f9d9-5dd5-4588-a1ea-8164055dff91 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:02.885206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 780/1553] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:02.885580 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69fa5b2-7bfb-4ecf-92cd-9d60d0bc552a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:02.886016 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa77a12-5676-4c4a-817d-8932d5a57816 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:02.886177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69fa5b2-7bfb-4ecf-92cd-9d60d0bc552a admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:02.886581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69fa5b2-7bfb-4ecf-92cd-9d60d0bc552a admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:02.886839 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69fa5b2-7bfb-4ecf-92cd-9d60d0bc552a admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:02.887414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69fa5b2-7bfb-4ecf-92cd-9d60d0bc552a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:02.889518 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69fa5b2-7bfb-4ecf-92cd-9d60d0bc552a admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:02.896389 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa77a12-5676-4c4a-817d-8932d5a57816 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:02.901479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35de0c9e-91ca-4f37-add9-4d6c7114b870 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:02.902089 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35de0c9e-91ca-4f37-add9-4d6c7114b870 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:02.902386 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35de0c9e-91ca-4f37-add9-4d6c7114b870 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:02.902656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35de0c9e-91ca-4f37-add9-4d6c7114b870 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:02.903460 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35de0c9e-91ca-4f37-add9-4d6c7114b870 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:02.904485 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: WARNING py.warnings [None req-35de0c9e-91ca-4f37-add9-4d6c7114b870 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 23 14:02:02.904485 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: warnings.warn(msg) Sep 23 14:02:02.904485 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: Sep 23 14:02:02.905684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35de0c9e-91ca-4f37-add9-4d6c7114b870 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:03.005111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 774/1554] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:02 2022] POST /identity/v3/users => generated 289 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:03.006317 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482439f3-9657-4901-8324-95a74af1a0ca None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.010962 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482439f3-9657-4901-8324-95a74af1a0ca None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.010962 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482439f3-9657-4901-8324-95a74af1a0ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.026943 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-482439f3-9657-4901-8324-95a74af1a0ca None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:03.028143 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-482439f3-9657-4901-8324-95a74af1a0ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:03.035652 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 781/1555] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:02 2022] POST /identity/v3/users => generated 297 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:03.035802 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e40309-2268-47bc-992c-4a35d57999d4 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.043854 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e40309-2268-47bc-992c-4a35d57999d4 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.043854 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e40309-2268-47bc-992c-4a35d57999d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.044084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 775/1556] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:03.044084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445b9972-88ce-409a-9a76-947b67fb0745 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.044084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445b9972-88ce-409a-9a76-947b67fb0745 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.044084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445b9972-88ce-409a-9a76-947b67fb0745 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.058981 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c3e40309-2268-47bc-992c-4a35d57999d4 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:03.062678 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c3e40309-2268-47bc-992c-4a35d57999d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:03.062819 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-445b9972-88ce-409a-9a76-947b67fb0745 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:03.063304 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-445b9972-88ce-409a-9a76-947b67fb0745 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:03.069254 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 782/1557] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:03.072328 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 776/1558] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:03.072968 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cf55f3-5f74-4a14-a6b8-ec2dd765d5f5 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.073445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cf55f3-5f74-4a14-a6b8-ec2dd765d5f5 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.073741 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44264562-6097-46d4-aa6d-c2fc8a347939 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.074084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44264562-6097-46d4-aa6d-c2fc8a347939 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.074419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cf55f3-5f74-4a14-a6b8-ec2dd765d5f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.074549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44264562-6097-46d4-aa6d-c2fc8a347939 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.093906 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-44264562-6097-46d4-aa6d-c2fc8a347939 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:03.095136 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-44264562-6097-46d4-aa6d-c2fc8a347939 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:03.095440 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-34cf55f3-5f74-4a14-a6b8-ec2dd765d5f5 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:03.096605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-34cf55f3-5f74-4a14-a6b8-ec2dd765d5f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:03.103277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 777/1559] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:03.104416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 783/1560] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:03.105181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34cf55f3-5f74-4a14-a6b8-ec2dd765d5f5 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:03.108821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275c00c6-8e6b-46fd-a920-fa062999ce13 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.109107 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275c00c6-8e6b-46fd-a920-fa062999ce13 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.109378 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275c00c6-8e6b-46fd-a920-fa062999ce13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.114704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34cf55f3-5f74-4a14-a6b8-ec2dd765d5f5 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:03.119883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ea2926-2797-4377-aace-a4431d7346eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:03.120396 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ea2926-2797-4377-aace-a4431d7346eb admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.120676 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ea2926-2797-4377-aace-a4431d7346eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.120943 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ea2926-2797-4377-aace-a4431d7346eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.121589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ea2926-2797-4377-aace-a4431d7346eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:03.123238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ea2926-2797-4377-aace-a4431d7346eb admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:03.124602 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:03.130747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-275c00c6-8e6b-46fd-a920-fa062999ce13 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:03.131935 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-275c00c6-8e6b-46fd-a920-fa062999ce13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:03.134839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 784/1561] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:03.135738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ea2926-2797-4377-aace-a4431d7346eb admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:03.141251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 778/1562] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:03.142889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275c00c6-8e6b-46fd-a920-fa062999ce13 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:03.145878 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ea2926-2797-4377-aace-a4431d7346eb admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:03.150822 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4f8224-5be6-4961-bc76-35fb678f620a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:03.151512 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4f8224-5be6-4961-bc76-35fb678f620a admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.151787 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4f8224-5be6-4961-bc76-35fb678f620a admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.152057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4f8224-5be6-4961-bc76-35fb678f620a admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.152688 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4f8224-5be6-4961-bc76-35fb678f620a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:03.153366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275c00c6-8e6b-46fd-a920-fa062999ce13 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:03.154418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4f8224-5be6-4961-bc76-35fb678f620a admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:03.158121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca19d397-66d6-428a-af4f-9b5e39f14daf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:03.158749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca19d397-66d6-428a-af4f-9b5e39f14daf admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.159024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca19d397-66d6-428a-af4f-9b5e39f14daf admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.159292 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca19d397-66d6-428a-af4f-9b5e39f14daf admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.160091 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca19d397-66d6-428a-af4f-9b5e39f14daf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:03.160631 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca19d397-66d6-428a-af4f-9b5e39f14daf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:03.161953 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca19d397-66d6-428a-af4f-9b5e39f14daf admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:03.165134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 785/1563] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:03.166060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4f8224-5be6-4961-bc76-35fb678f620a admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:03.173815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 779/1564] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:02:03.174642 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca19d397-66d6-428a-af4f-9b5e39f14daf admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:03.175588 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4f8224-5be6-4961-bc76-35fb678f620a admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:03.181915 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af3d00d-2317-405d-a46a-3cec193d7a3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:03.182744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af3d00d-2317-405d-a46a-3cec193d7a3d admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.183200 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af3d00d-2317-405d-a46a-3cec193d7a3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.183713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af3d00d-2317-405d-a46a-3cec193d7a3d admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.184353 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af3d00d-2317-405d-a46a-3cec193d7a3d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:03.184651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af3d00d-2317-405d-a46a-3cec193d7a3d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:03.186259 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af3d00d-2317-405d-a46a-3cec193d7a3d admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:03.190714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca19d397-66d6-428a-af4f-9b5e39f14daf admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:03.195212 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c460fa3-8c26-48bd-a6be-91931d019848 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:03.197901 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c460fa3-8c26-48bd-a6be-91931d019848 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.198198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c460fa3-8c26-48bd-a6be-91931d019848 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.202256 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c460fa3-8c26-48bd-a6be-91931d019848 admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.202452 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 786/1565] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:02:03.202566 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c460fa3-8c26-48bd-a6be-91931d019848 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:03.202566 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c460fa3-8c26-48bd-a6be-91931d019848 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:03.202566 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c460fa3-8c26-48bd-a6be-91931d019848 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:03.202823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af3d00d-2317-405d-a46a-3cec193d7a3d admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:03.213152 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af3d00d-2317-405d-a46a-3cec193d7a3d admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:03.218659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dc2bf9-5c9c-4440-ab80-1a842a2a5db7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:03.219205 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 780/1566] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:02:03.219614 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dc2bf9-5c9c-4440-ab80-1a842a2a5db7 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.219897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dc2bf9-5c9c-4440-ab80-1a842a2a5db7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.220167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15dc2bf9-5c9c-4440-ab80-1a842a2a5db7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.220431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c460fa3-8c26-48bd-a6be-91931d019848 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:03.220822 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dc2bf9-5c9c-4440-ab80-1a842a2a5db7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:03.221121 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dc2bf9-5c9c-4440-ab80-1a842a2a5db7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:03.222746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15dc2bf9-5c9c-4440-ab80-1a842a2a5db7 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:03.231360 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c460fa3-8c26-48bd-a6be-91931d019848 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:03.236025 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6aa8009-f3ec-4029-ad15-3f02775f067e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:03.236616 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aa8009-f3ec-4029-ad15-3f02775f067e admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.236891 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aa8009-f3ec-4029-ad15-3f02775f067e admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.237157 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aa8009-f3ec-4029-ad15-3f02775f067e admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.237791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6aa8009-f3ec-4029-ad15-3f02775f067e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:03.238090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6aa8009-f3ec-4029-ad15-3f02775f067e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:03.245847 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 787/1567] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:02:03.246039 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6aa8009-f3ec-4029-ad15-3f02775f067e admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:03.246144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dc2bf9-5c9c-4440-ab80-1a842a2a5db7 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:03.249939 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15dc2bf9-5c9c-4440-ab80-1a842a2a5db7 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:03.252112 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 781/1568] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:02:03.252925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6aa8009-f3ec-4029-ad15-3f02775f067e admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:03.260548 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4cc6ef-347e-48ac-b33e-87b7289f1624 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:03.261095 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4cc6ef-347e-48ac-b33e-87b7289f1624 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.261373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4cc6ef-347e-48ac-b33e-87b7289f1624 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.261645 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4cc6ef-347e-48ac-b33e-87b7289f1624 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.262285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4cc6ef-347e-48ac-b33e-87b7289f1624 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:03.263912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4cc6ef-347e-48ac-b33e-87b7289f1624 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:03.267695 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6aa8009-f3ec-4029-ad15-3f02775f067e admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:03.272598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec59cef3-3204-40c4-8506-65eb58a74b0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:03.273185 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec59cef3-3204-40c4-8506-65eb58a74b0a admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.273459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec59cef3-3204-40c4-8506-65eb58a74b0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.274396 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec59cef3-3204-40c4-8506-65eb58a74b0a admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.275035 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec59cef3-3204-40c4-8506-65eb58a74b0a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:03.275344 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec59cef3-3204-40c4-8506-65eb58a74b0a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:03.279416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 788/1569] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:03 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:03.279416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4cc6ef-347e-48ac-b33e-87b7289f1624 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:03.279730 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec59cef3-3204-40c4-8506-65eb58a74b0a admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:03.286034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4cc6ef-347e-48ac-b33e-87b7289f1624 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:03.291360 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 782/1570] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:02:03.291625 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300e12a6-b07c-481d-8702-f5bd5c707c78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:03.292226 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300e12a6-b07c-481d-8702-f5bd5c707c78 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.292435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec59cef3-3204-40c4-8506-65eb58a74b0a admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:03.292558 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300e12a6-b07c-481d-8702-f5bd5c707c78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.292760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300e12a6-b07c-481d-8702-f5bd5c707c78 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.293407 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300e12a6-b07c-481d-8702-f5bd5c707c78 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:03.295066 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300e12a6-b07c-481d-8702-f5bd5c707c78 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:03.302965 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:03.306184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 789/1571] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:03.307355 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300e12a6-b07c-481d-8702-f5bd5c707c78 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:03.308399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec59cef3-3204-40c4-8506-65eb58a74b0a admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:03.313880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0802e2f2-bdc5-4e63-abbc-0cb47ff967c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:03.314589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0802e2f2-bdc5-4e63-abbc-0cb47ff967c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.314867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0802e2f2-bdc5-4e63-abbc-0cb47ff967c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.315133 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0802e2f2-bdc5-4e63-abbc-0cb47ff967c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.315756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0802e2f2-bdc5-4e63-abbc-0cb47ff967c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:03.317063 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300e12a6-b07c-481d-8702-f5bd5c707c78 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:03.317642 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0802e2f2-bdc5-4e63-abbc-0cb47ff967c6 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:03.322679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd202977-2225-4692-bb90-044579ed896b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:03.323267 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd202977-2225-4692-bb90-044579ed896b admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.323543 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd202977-2225-4692-bb90-044579ed896b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.323838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd202977-2225-4692-bb90-044579ed896b admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.324435 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd202977-2225-4692-bb90-044579ed896b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:03.326313 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd202977-2225-4692-bb90-044579ed896b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:03.418640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 783/1572] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:03 2022] POST /identity/v3/projects => generated 379 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:03.419478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0802e2f2-bdc5-4e63-abbc-0cb47ff967c6 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:03.431088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 790/1573] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:03 2022] POST /identity/v3/projects => generated 385 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:03.432102 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd202977-2225-4692-bb90-044579ed896b admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:03.483444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0802e2f2-bdc5-4e63-abbc-0cb47ff967c6 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:03.497482 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd202977-2225-4692-bb90-044579ed896b admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:03.502371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c47e222-cdd8-4b3f-a2db-5bc018fec040 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:03.502971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c47e222-cdd8-4b3f-a2db-5bc018fec040 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.503246 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c47e222-cdd8-4b3f-a2db-5bc018fec040 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.503512 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c47e222-cdd8-4b3f-a2db-5bc018fec040 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.504133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c47e222-cdd8-4b3f-a2db-5bc018fec040 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:03.506165 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c47e222-cdd8-4b3f-a2db-5bc018fec040 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:03.541908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f675be2-fb7c-4c17-8cc5-94f9ae19c699 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:03.542474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f675be2-fb7c-4c17-8cc5-94f9ae19c699 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.542757 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f675be2-fb7c-4c17-8cc5-94f9ae19c699 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.543030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f675be2-fb7c-4c17-8cc5-94f9ae19c699 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.543660 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f675be2-fb7c-4c17-8cc5-94f9ae19c699 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:03.545556 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f675be2-fb7c-4c17-8cc5-94f9ae19c699 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:03.606899 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 791/1574] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:03 2022] POST /identity/v3/projects => generated 405 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:03.608472 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c47e222-cdd8-4b3f-a2db-5bc018fec040 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:03.636623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 784/1575] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:03 2022] POST /identity/v3/projects => generated 365 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:03.638898 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f675be2-fb7c-4c17-8cc5-94f9ae19c699 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:03.673525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c47e222-cdd8-4b3f-a2db-5bc018fec040 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:03.677075 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:03.701869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f675be2-fb7c-4c17-8cc5-94f9ae19c699 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:03.706678 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec44476-8053-4c1f-9663-35a385f4d489 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:03.707377 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec44476-8053-4c1f-9663-35a385f4d489 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.707659 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec44476-8053-4c1f-9663-35a385f4d489 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.707930 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec44476-8053-4c1f-9663-35a385f4d489 admin admin] PATH_INFO: `/v3/projects/265cc577b49941f88bf16b2475f76f3b/users/242bf7d8bba64480814d8194ea12b19a/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.722557 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec44476-8053-4c1f-9663-35a385f4d489 admin admin] RBAC: Authorizing `identity:create_grant(project_id=265cc577b49941f88bf16b2475f76f3b, user_id=242bf7d8bba64480814d8194ea12b19a, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:03.723574 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: WARNING py.warnings [None req-1ec44476-8053-4c1f-9663-35a385f4d489 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 23 14:02:03.723574 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: warnings.warn(msg) Sep 23 14:02:03.723574 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: Sep 23 14:02:03.725550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec44476-8053-4c1f-9663-35a385f4d489 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:03.729015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:03.733660 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf1b98a-38e8-49a6-8a5b-4af46943b46e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:03.735206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf1b98a-38e8-49a6-8a5b-4af46943b46e admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.735206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf1b98a-38e8-49a6-8a5b-4af46943b46e admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.735206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf1b98a-38e8-49a6-8a5b-4af46943b46e admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.735659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf1b98a-38e8-49a6-8a5b-4af46943b46e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:03.737566 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf1b98a-38e8-49a6-8a5b-4af46943b46e admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:03.832978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 785/1576] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:03 2022] PUT /identity/v3/projects/265cc577b49941f88bf16b2475f76f3b/users/242bf7d8bba64480814d8194ea12b19a/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:03.833704 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec44476-8053-4c1f-9663-35a385f4d489 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:03.857909 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 792/1577] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:03 2022] POST /identity/v3/projects => generated 377 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:03.857909 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf1b98a-38e8-49a6-8a5b-4af46943b46e admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:03.896654 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec44476-8053-4c1f-9663-35a385f4d489 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:03.916432 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf1b98a-38e8-49a6-8a5b-4af46943b46e admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:03.921104 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a59ef73-0f79-42a0-bba5-ba3b568cb305 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:03.921762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a59ef73-0f79-42a0-bba5-ba3b568cb305 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.922040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a59ef73-0f79-42a0-bba5-ba3b568cb305 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.922308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a59ef73-0f79-42a0-bba5-ba3b568cb305 admin admin] PATH_INFO: `/v3/projects/d08a219027684089a91c354749c67d5d/users/28b64a35d7b44519a1d93919f2e23083/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.939254 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a59ef73-0f79-42a0-bba5-ba3b568cb305 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d08a219027684089a91c354749c67d5d, user_id=28b64a35d7b44519a1d93919f2e23083, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:03.942063 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a59ef73-0f79-42a0-bba5-ba3b568cb305 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:03.954028 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42fc58c-13f9-438a-8928-9a93e9d4f289 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:03.956969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42fc58c-13f9-438a-8928-9a93e9d4f289 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:03.956969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42fc58c-13f9-438a-8928-9a93e9d4f289 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:03.956969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e42fc58c-13f9-438a-8928-9a93e9d4f289 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:03.956969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e42fc58c-13f9-438a-8928-9a93e9d4f289 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:03.958539 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e42fc58c-13f9-438a-8928-9a93e9d4f289 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.009986 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 793/1578] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:03 2022] PUT /identity/v3/projects/d08a219027684089a91c354749c67d5d/users/28b64a35d7b44519a1d93919f2e23083/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:04.010962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a59ef73-0f79-42a0-bba5-ba3b568cb305 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.042791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 786/1579] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:03 2022] POST /identity/v3/projects => generated 373 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:04.042791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42fc58c-13f9-438a-8928-9a93e9d4f289 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.077025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a59ef73-0f79-42a0-bba5-ba3b568cb305 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.097380 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:04.106197 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e42fc58c-13f9-438a-8928-9a93e9d4f289 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.110940 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7800f2d8-2670-439b-8bce-0be90db48023 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.111543 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7800f2d8-2670-439b-8bce-0be90db48023 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.111821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7800f2d8-2670-439b-8bce-0be90db48023 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.112091 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7800f2d8-2670-439b-8bce-0be90db48023 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.112779 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7800f2d8-2670-439b-8bce-0be90db48023 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.114723 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7800f2d8-2670-439b-8bce-0be90db48023 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.136394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df74ac4c-5279-4d85-bdfb-31552f813f25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.136996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df74ac4c-5279-4d85-bdfb-31552f813f25 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.137270 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df74ac4c-5279-4d85-bdfb-31552f813f25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.137539 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df74ac4c-5279-4d85-bdfb-31552f813f25 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.138159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df74ac4c-5279-4d85-bdfb-31552f813f25 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.140095 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df74ac4c-5279-4d85-bdfb-31552f813f25 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.205868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 787/1580] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:04 2022] POST /identity/v3/users => generated 293 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:04.206713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7800f2d8-2670-439b-8bce-0be90db48023 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.218274 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 794/1581] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:04 2022] POST /identity/v3/users => generated 290 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:04.218771 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7800f2d8-2670-439b-8bce-0be90db48023 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.219425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df74ac4c-5279-4d85-bdfb-31552f813f25 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.225776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b67391-3f24-4404-b008-2b461922da36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.226423 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b67391-3f24-4404-b008-2b461922da36 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.227251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b67391-3f24-4404-b008-2b461922da36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.227500 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b67391-3f24-4404-b008-2b461922da36 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.228452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b67391-3f24-4404-b008-2b461922da36 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.229974 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df74ac4c-5279-4d85-bdfb-31552f813f25 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.230414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b67391-3f24-4404-b008-2b461922da36 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.234562 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e146697-f035-4000-b2ba-c1d4f804075f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.235148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e146697-f035-4000-b2ba-c1d4f804075f admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.235422 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e146697-f035-4000-b2ba-c1d4f804075f admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.235689 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e146697-f035-4000-b2ba-c1d4f804075f admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.236300 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e146697-f035-4000-b2ba-c1d4f804075f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.238173 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e146697-f035-4000-b2ba-c1d4f804075f admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.315316 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:04.321278 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 788/1582] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:04 2022] POST /identity/v3/users => generated 303 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:04.322462 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132e473a-7b37-49af-b784-a3099b63c42c None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.322848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132e473a-7b37-49af-b784-a3099b63c42c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.323128 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-132e473a-7b37-49af-b784-a3099b63c42c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.336509 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 795/1583] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:04 2022] POST /identity/v3/users => generated 283 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:04.337449 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e146697-f035-4000-b2ba-c1d4f804075f admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.347698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e146697-f035-4000-b2ba-c1d4f804075f admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.351017 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-132e473a-7b37-49af-b784-a3099b63c42c None None] MFA Rules not processed for user `242bf7d8bba64480814d8194ea12b19a`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:04.352894 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-132e473a-7b37-49af-b784-a3099b63c42c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:04.353159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588effdc-f523-4484-b1db-a9e5fff573c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.353765 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588effdc-f523-4484-b1db-a9e5fff573c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.354041 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588effdc-f523-4484-b1db-a9e5fff573c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.354382 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588effdc-f523-4484-b1db-a9e5fff573c5 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.355353 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588effdc-f523-4484-b1db-a9e5fff573c5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.357760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588effdc-f523-4484-b1db-a9e5fff573c5 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.420913 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 789/1584] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:04 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:04.422155 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2771d116-2970-49a3-bdab-79b53be025b8 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.422468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2771d116-2970-49a3-bdab-79b53be025b8 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.422742 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2771d116-2970-49a3-bdab-79b53be025b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.441987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-2771d116-2970-49a3-bdab-79b53be025b8 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:04.443215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2771d116-2970-49a3-bdab-79b53be025b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:04.451746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 796/1585] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:04 2022] POST /identity/v3/users => generated 289 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:04.452603 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588effdc-f523-4484-b1db-a9e5fff573c5 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.454791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 790/1586] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:04.456097 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2771d116-2970-49a3-bdab-79b53be025b8 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.465966 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588effdc-f523-4484-b1db-a9e5fff573c5 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.468586 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2771d116-2970-49a3-bdab-79b53be025b8 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.470731 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b3e131-cf2d-47be-aa0b-298c6ae904e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.471333 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b3e131-cf2d-47be-aa0b-298c6ae904e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.471609 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b3e131-cf2d-47be-aa0b-298c6ae904e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.471875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b3e131-cf2d-47be-aa0b-298c6ae904e2 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.472503 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b3e131-cf2d-47be-aa0b-298c6ae904e2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.473543 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8615df-e37b-465a-98e6-fc2bc4c06170 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.474260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8615df-e37b-465a-98e6-fc2bc4c06170 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.474430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b3e131-cf2d-47be-aa0b-298c6ae904e2 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.474596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8615df-e37b-465a-98e6-fc2bc4c06170 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.476738 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8615df-e37b-465a-98e6-fc2bc4c06170 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.476738 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8615df-e37b-465a-98e6-fc2bc4c06170 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.478321 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8615df-e37b-465a-98e6-fc2bc4c06170 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.489276 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 791/1587] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:04.490079 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8615df-e37b-465a-98e6-fc2bc4c06170 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.500466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8615df-e37b-465a-98e6-fc2bc4c06170 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.505327 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfee5a1c-5164-45cf-b098-6525ed1b1edb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.505668 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfee5a1c-5164-45cf-b098-6525ed1b1edb admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.505948 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfee5a1c-5164-45cf-b098-6525ed1b1edb admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.506222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfee5a1c-5164-45cf-b098-6525ed1b1edb admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.506889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfee5a1c-5164-45cf-b098-6525ed1b1edb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.508927 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfee5a1c-5164-45cf-b098-6525ed1b1edb admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.518737 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 792/1588] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:04.519510 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfee5a1c-5164-45cf-b098-6525ed1b1edb admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.529689 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfee5a1c-5164-45cf-b098-6525ed1b1edb admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.537358 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a94c81f-4d8b-469e-8eb5-af66eb6e92cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.537358 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a94c81f-4d8b-469e-8eb5-af66eb6e92cf admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.537358 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a94c81f-4d8b-469e-8eb5-af66eb6e92cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.537358 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a94c81f-4d8b-469e-8eb5-af66eb6e92cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.537752 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a94c81f-4d8b-469e-8eb5-af66eb6e92cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.537752 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a94c81f-4d8b-469e-8eb5-af66eb6e92cf admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.548418 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 793/1589] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:04.549254 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a94c81f-4d8b-469e-8eb5-af66eb6e92cf admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.559595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a94c81f-4d8b-469e-8eb5-af66eb6e92cf admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.560027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 797/1590] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:04 2022] POST /identity/v3/users => generated 287 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:04.562081 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b3e131-cf2d-47be-aa0b-298c6ae904e2 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.564366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf24fe03-b7ae-456e-ba9d-96a5741ab90d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.568745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf24fe03-b7ae-456e-ba9d-96a5741ab90d admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.568745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf24fe03-b7ae-456e-ba9d-96a5741ab90d admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.568745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf24fe03-b7ae-456e-ba9d-96a5741ab90d admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.568745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf24fe03-b7ae-456e-ba9d-96a5741ab90d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.568745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf24fe03-b7ae-456e-ba9d-96a5741ab90d admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.572168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b3e131-cf2d-47be-aa0b-298c6ae904e2 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.576833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a02dea4-be57-4e66-bf86-e84d7f7bd617 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.577406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a02dea4-be57-4e66-bf86-e84d7f7bd617 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.577686 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a02dea4-be57-4e66-bf86-e84d7f7bd617 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.577958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a02dea4-be57-4e66-bf86-e84d7f7bd617 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.578615 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a02dea4-be57-4e66-bf86-e84d7f7bd617 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.582210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 794/1591] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:04.582210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf24fe03-b7ae-456e-ba9d-96a5741ab90d admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.582486 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a02dea4-be57-4e66-bf86-e84d7f7bd617 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.592264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf24fe03-b7ae-456e-ba9d-96a5741ab90d admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.597101 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793c19ff-64b4-402f-a127-e05e02d3c58c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.597928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793c19ff-64b4-402f-a127-e05e02d3c58c admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.598634 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793c19ff-64b4-402f-a127-e05e02d3c58c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.598913 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793c19ff-64b4-402f-a127-e05e02d3c58c admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.599614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793c19ff-64b4-402f-a127-e05e02d3c58c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.601178 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793c19ff-64b4-402f-a127-e05e02d3c58c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.612109 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 795/1592] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:04.616583 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793c19ff-64b4-402f-a127-e05e02d3c58c admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.665296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793c19ff-64b4-402f-a127-e05e02d3c58c admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.668907 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf4a618-05ad-4d41-aeef-ea8458356b0d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.669483 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf4a618-05ad-4d41-aeef-ea8458356b0d service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.669760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf4a618-05ad-4d41-aeef-ea8458356b0d service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.670030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf4a618-05ad-4d41-aeef-ea8458356b0d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.678867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf4a618-05ad-4d41-aeef-ea8458356b0d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.683444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf4a618-05ad-4d41-aeef-ea8458356b0d service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.686915 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 798/1593] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:04 2022] POST /identity/v3/projects => generated 375 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:04.687735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a02dea4-be57-4e66-bf86-e84d7f7bd617 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.690950 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 796/1594] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:04.691974 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf4a618-05ad-4d41-aeef-ea8458356b0d service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.701834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf4a618-05ad-4d41-aeef-ea8458356b0d service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.733284 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:04.752021 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a02dea4-be57-4e66-bf86-e84d7f7bd617 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.756605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b767336f-72d5-4c16-9b8e-4539316d5693 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.757204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b767336f-72d5-4c16-9b8e-4539316d5693 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.757506 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b767336f-72d5-4c16-9b8e-4539316d5693 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.757778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b767336f-72d5-4c16-9b8e-4539316d5693 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.758972 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b767336f-72d5-4c16-9b8e-4539316d5693 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.760435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99ea452-d259-41ee-a0db-a82d228532b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.760575 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b767336f-72d5-4c16-9b8e-4539316d5693 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.761040 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99ea452-d259-41ee-a0db-a82d228532b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.761318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99ea452-d259-41ee-a0db-a82d228532b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.761584 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99ea452-d259-41ee-a0db-a82d228532b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.762215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99ea452-d259-41ee-a0db-a82d228532b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.763806 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99ea452-d259-41ee-a0db-a82d228532b8 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.770674 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 799/1595] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:04.773484 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 797/1596] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:04.775318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99ea452-d259-41ee-a0db-a82d228532b8 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.775457 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b767336f-72d5-4c16-9b8e-4539316d5693 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.784697 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99ea452-d259-41ee-a0db-a82d228532b8 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.787223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b767336f-72d5-4c16-9b8e-4539316d5693 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.789677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e83c61-ea04-40f4-87b7-ff171abd8874 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.790373 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e83c61-ea04-40f4-87b7-ff171abd8874 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.790660 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e83c61-ea04-40f4-87b7-ff171abd8874 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.790926 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e83c61-ea04-40f4-87b7-ff171abd8874 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.791607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e83c61-ea04-40f4-87b7-ff171abd8874 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.792484 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77d7d6b-20bc-4c41-83c1-513eed61e254 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.793224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77d7d6b-20bc-4c41-83c1-513eed61e254 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.793333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e83c61-ea04-40f4-87b7-ff171abd8874 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.793412 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77d7d6b-20bc-4c41-83c1-513eed61e254 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.793608 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77d7d6b-20bc-4c41-83c1-513eed61e254 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.794522 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77d7d6b-20bc-4c41-83c1-513eed61e254 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.798122 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77d7d6b-20bc-4c41-83c1-513eed61e254 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.803550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 798/1597] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:04.807533 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e83c61-ea04-40f4-87b7-ff171abd8874 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.811126 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 800/1598] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:04.811126 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77d7d6b-20bc-4c41-83c1-513eed61e254 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.821893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77d7d6b-20bc-4c41-83c1-513eed61e254 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.822653 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e83c61-ea04-40f4-87b7-ff171abd8874 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.827108 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4cb127-52f0-4b5b-8c71-90017b06e5d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.827711 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4cb127-52f0-4b5b-8c71-90017b06e5d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.827956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281326ef-7a73-459b-8af0-1f20617c5ad5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.828082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4cb127-52f0-4b5b-8c71-90017b06e5d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.828266 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4cb127-52f0-4b5b-8c71-90017b06e5d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.828723 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281326ef-7a73-459b-8af0-1f20617c5ad5 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.828902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4cb127-52f0-4b5b-8c71-90017b06e5d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.829121 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281326ef-7a73-459b-8af0-1f20617c5ad5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.829441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281326ef-7a73-459b-8af0-1f20617c5ad5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.830140 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281326ef-7a73-459b-8af0-1f20617c5ad5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.830611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4cb127-52f0-4b5b-8c71-90017b06e5d2 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.831996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281326ef-7a73-459b-8af0-1f20617c5ad5 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.842381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 799/1599] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:04.845870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 801/1600] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:04 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:04.846764 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4cb127-52f0-4b5b-8c71-90017b06e5d2 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.852858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281326ef-7a73-459b-8af0-1f20617c5ad5 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:04.856706 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281326ef-7a73-459b-8af0-1f20617c5ad5 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.859165 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4cb127-52f0-4b5b-8c71-90017b06e5d2 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:04.861740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4661f68e-a8aa-4fb3-9d2b-5e197f0db350 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.862424 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4661f68e-a8aa-4fb3-9d2b-5e197f0db350 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.862697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4661f68e-a8aa-4fb3-9d2b-5e197f0db350 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.862959 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4661f68e-a8aa-4fb3-9d2b-5e197f0db350 admin admin] PATH_INFO: `/v3/projects/482490c45ac64607bba9ad021f2effbf/users/0db30f990aae49fabf9b8b19cd019de9/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.864681 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d29e0b-c86d-4066-a52c-cfea9dd4e59c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:04.865432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d29e0b-c86d-4066-a52c-cfea9dd4e59c admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:04.865773 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d29e0b-c86d-4066-a52c-cfea9dd4e59c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:04.866110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d29e0b-c86d-4066-a52c-cfea9dd4e59c admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:04.866939 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d29e0b-c86d-4066-a52c-cfea9dd4e59c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.868972 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d29e0b-c86d-4066-a52c-cfea9dd4e59c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.872928 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=87,keystone:INSERT=13 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:04.880543 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4661f68e-a8aa-4fb3-9d2b-5e197f0db350 admin admin] RBAC: Authorizing `identity:create_grant(project_id=482490c45ac64607bba9ad021f2effbf, user_id=0db30f990aae49fabf9b8b19cd019de9, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:04.884239 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4661f68e-a8aa-4fb3-9d2b-5e197f0db350 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:04.974978 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 802/1601] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:04 2022] PUT /identity/v3/projects/482490c45ac64607bba9ad021f2effbf/users/0db30f990aae49fabf9b8b19cd019de9/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:04.975860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4661f68e-a8aa-4fb3-9d2b-5e197f0db350 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:05.012536 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 800/1602] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:04 2022] POST /identity/v3/users => generated 294 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:05.016544 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d29e0b-c86d-4066-a52c-cfea9dd4e59c admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:05.025245 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d29e0b-c86d-4066-a52c-cfea9dd4e59c admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:05.042879 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4661f68e-a8aa-4fb3-9d2b-5e197f0db350 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:05.049357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a775cee3-ae2c-468c-8eb9-ce721b5afd7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:05.050625 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a775cee3-ae2c-468c-8eb9-ce721b5afd7c admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:05.050911 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a775cee3-ae2c-468c-8eb9-ce721b5afd7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:05.051183 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a775cee3-ae2c-468c-8eb9-ce721b5afd7c admin admin] PATH_INFO: `/v3/projects/fd25990a10694a5aa7debfcae216d00c/users/6d5d064339d14cd09569765c3ce9b0f3/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:05.071954 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a775cee3-ae2c-468c-8eb9-ce721b5afd7c admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd25990a10694a5aa7debfcae216d00c, user_id=6d5d064339d14cd09569765c3ce9b0f3, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:05.074703 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a775cee3-ae2c-468c-8eb9-ce721b5afd7c admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:05.093457 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34087869-ee68-49ec-8862-decb3904c825 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:05.094078 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34087869-ee68-49ec-8862-decb3904c825 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:05.094388 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34087869-ee68-49ec-8862-decb3904c825 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:05.094666 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34087869-ee68-49ec-8862-decb3904c825 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:05.095313 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34087869-ee68-49ec-8862-decb3904c825 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:05.096914 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34087869-ee68-49ec-8862-decb3904c825 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:05.108096 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 801/1603] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:05 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:05.108856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34087869-ee68-49ec-8862-decb3904c825 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:05.119337 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34087869-ee68-49ec-8862-decb3904c825 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:05.126880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8c8876-d053-493f-b816-10ba849f79b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:05.126880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8c8876-d053-493f-b816-10ba849f79b4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:05.127204 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8c8876-d053-493f-b816-10ba849f79b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:05.127586 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8c8876-d053-493f-b816-10ba849f79b4 admin admin] PATH_INFO: `/v3/projects/b5ba4efc80634f739469cfdeb8a5e96f/users/bb36a6c7bf3945a39f01605ff5e0af75/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:05.145549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8c8876-d053-493f-b816-10ba849f79b4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5ba4efc80634f739469cfdeb8a5e96f, user_id=bb36a6c7bf3945a39f01605ff5e0af75, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:05.148329 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8c8876-d053-493f-b816-10ba849f79b4 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:05.160449 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 803/1604] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:04 2022] PUT /identity/v3/projects/fd25990a10694a5aa7debfcae216d00c/users/6d5d064339d14cd09569765c3ce9b0f3/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:05.161340 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a775cee3-ae2c-468c-8eb9-ce721b5afd7c admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:05.225113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a775cee3-ae2c-468c-8eb9-ce721b5afd7c admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:05.230649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7feb1641-b147-4442-8157-54145ed9de56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:05.231303 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feb1641-b147-4442-8157-54145ed9de56 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:05.231591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feb1641-b147-4442-8157-54145ed9de56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:05.231939 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7feb1641-b147-4442-8157-54145ed9de56 admin admin] PATH_INFO: `/v3/projects/4a25811b83194cbf820eb25171f370a6/users/2687ba3a77d64bb99bbe6bd38b9b63a2/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:05.246042 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 802/1605] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:05 2022] PUT /identity/v3/projects/b5ba4efc80634f739469cfdeb8a5e96f/users/bb36a6c7bf3945a39f01605ff5e0af75/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:05.248771 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8c8876-d053-493f-b816-10ba849f79b4 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:05.253027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7feb1641-b147-4442-8157-54145ed9de56 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4a25811b83194cbf820eb25171f370a6, user_id=2687ba3a77d64bb99bbe6bd38b9b63a2, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:05.255781 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7feb1641-b147-4442-8157-54145ed9de56 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:05.320394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8c8876-d053-493f-b816-10ba849f79b4 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:05.325453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb51ead-7aaf-4618-84cf-7d2e8226296f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:05.326358 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb51ead-7aaf-4618-84cf-7d2e8226296f admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:05.326971 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb51ead-7aaf-4618-84cf-7d2e8226296f admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:05.327252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb51ead-7aaf-4618-84cf-7d2e8226296f admin admin] PATH_INFO: `/v3/projects/8e1a67a82fb44cf59f9b4f30848881d1/users/88a8db8d7d494c62b65e427278ef9a2a/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:05.330015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:05.332534 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 804/1606] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:05 2022] PUT /identity/v3/projects/4a25811b83194cbf820eb25171f370a6/users/2687ba3a77d64bb99bbe6bd38b9b63a2/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:05.346139 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab73bb3-6bba-4167-9d47-9fa1bf9c10eb None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:05.346526 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab73bb3-6bba-4167-9d47-9fa1bf9c10eb None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:05.346803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab73bb3-6bba-4167-9d47-9fa1bf9c10eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:05.354781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb51ead-7aaf-4618-84cf-7d2e8226296f admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e1a67a82fb44cf59f9b4f30848881d1, user_id=88a8db8d7d494c62b65e427278ef9a2a, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:05.363682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb51ead-7aaf-4618-84cf-7d2e8226296f admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:05.367309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-6ab73bb3-6bba-4167-9d47-9fa1bf9c10eb None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:05.368529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-6ab73bb3-6bba-4167-9d47-9fa1bf9c10eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:05.420119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:05.432808 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 805/1607] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:05.433816 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab73bb3-6bba-4167-9d47-9fa1bf9c10eb None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:05.443356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab73bb3-6bba-4167-9d47-9fa1bf9c10eb None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:05.448425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702f1901-03d5-4237-a9e1-0bd772191bd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:05.449074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702f1901-03d5-4237-a9e1-0bd772191bd4 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:05.449352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702f1901-03d5-4237-a9e1-0bd772191bd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:05.449663 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702f1901-03d5-4237-a9e1-0bd772191bd4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:05.450355 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702f1901-03d5-4237-a9e1-0bd772191bd4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:05.455389 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702f1901-03d5-4237-a9e1-0bd772191bd4 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:05.464379 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 806/1608] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:05 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:05.465328 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702f1901-03d5-4237-a9e1-0bd772191bd4 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:05.475432 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702f1901-03d5-4237-a9e1-0bd772191bd4 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:05.480517 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 803/1609] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:05 2022] PUT /identity/v3/projects/8e1a67a82fb44cf59f9b4f30848881d1/users/88a8db8d7d494c62b65e427278ef9a2a/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:05.482835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918ce88b-cbfc-4f3c-9cbd-1abcfb33f7c6 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:05.482835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918ce88b-cbfc-4f3c-9cbd-1abcfb33f7c6 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:05.482835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918ce88b-cbfc-4f3c-9cbd-1abcfb33f7c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:05.511758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-918ce88b-cbfc-4f3c-9cbd-1abcfb33f7c6 None None] MFA Rules not processed for user `6d5d064339d14cd09569765c3ce9b0f3`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:05.513158 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-918ce88b-cbfc-4f3c-9cbd-1abcfb33f7c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:05.541338 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa9975f-f3d2-477b-bd65-7ca2391c3acc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:05.542007 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa9975f-f3d2-477b-bd65-7ca2391c3acc admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:05.542290 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa9975f-f3d2-477b-bd65-7ca2391c3acc admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:05.542703 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa9975f-f3d2-477b-bd65-7ca2391c3acc admin admin] PATH_INFO: `/v3/projects/23b284ca6c134f958ca475088e669dec/users/58f69d8a60a14a2ea87fd840e8b0c916/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:05.560891 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa9975f-f3d2-477b-bd65-7ca2391c3acc admin admin] RBAC: Authorizing `identity:create_grant(project_id=23b284ca6c134f958ca475088e669dec, user_id=58f69d8a60a14a2ea87fd840e8b0c916, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:05.563741 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa9975f-f3d2-477b-bd65-7ca2391c3acc admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:05.583497 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 804/1610] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:05 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:05.584759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3c91c1-2946-4b81-9f50-78ad84af157a None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:05.585047 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3c91c1-2946-4b81-9f50-78ad84af157a None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:05.585323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3c91c1-2946-4b81-9f50-78ad84af157a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:05.614709 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-6c3c91c1-2946-4b81-9f50-78ad84af157a None None] MFA Rules not processed for user `bb36a6c7bf3945a39f01605ff5e0af75`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:05.615883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-6c3c91c1-2946-4b81-9f50-78ad84af157a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:05.663736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 807/1611] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:05 2022] PUT /identity/v3/projects/23b284ca6c134f958ca475088e669dec/users/58f69d8a60a14a2ea87fd840e8b0c916/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:05.666088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2506bc96-40ee-4fd8-9e79-ea4a1d15b342 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:05.666527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2506bc96-40ee-4fd8-9e79-ea4a1d15b342 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:05.666952 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2506bc96-40ee-4fd8-9e79-ea4a1d15b342 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:05.687025 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 805/1612] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:05 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:05.690740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3c91c1-2946-4b81-9f50-78ad84af157a None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:05.697988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-2506bc96-40ee-4fd8-9e79-ea4a1d15b342 None None] MFA Rules not processed for user `2687ba3a77d64bb99bbe6bd38b9b63a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:05.699212 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-2506bc96-40ee-4fd8-9e79-ea4a1d15b342 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:05.748455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3c91c1-2946-4b81-9f50-78ad84af157a None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:05.762482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:05.770144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 808/1613] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:05 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:05.770144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2506bc96-40ee-4fd8-9e79-ea4a1d15b342 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:05.801791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d268b8-eb44-4239-b2e4-972d8c348868 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:05.802399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d268b8-eb44-4239-b2e4-972d8c348868 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:05.803132 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d268b8-eb44-4239-b2e4-972d8c348868 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:05.803132 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d268b8-eb44-4239-b2e4-972d8c348868 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:05.831708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:05.838599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2506bc96-40ee-4fd8-9e79-ea4a1d15b342 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:05.845289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e207227-5301-441a-a682-b0a7c5c2fceb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:05.846166 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e207227-5301-441a-a682-b0a7c5c2fceb admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:05.846733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e207227-5301-441a-a682-b0a7c5c2fceb admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:05.847108 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e207227-5301-441a-a682-b0a7c5c2fceb admin admin] PATH_INFO: `/v3/projects/2353a6dd9ae24027ae79623008a5d979/users/d675c12bb79f4c95887529e6b26f00ad/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:05.896405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e207227-5301-441a-a682-b0a7c5c2fceb admin admin] RBAC: Authorizing `identity:create_grant(project_id=2353a6dd9ae24027ae79623008a5d979, user_id=d675c12bb79f4c95887529e6b26f00ad, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:05.899082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e207227-5301-441a-a682-b0a7c5c2fceb admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:05.918710 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d268b8-eb44-4239-b2e4-972d8c348868 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:05.920123 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d268b8-eb44-4239-b2e4-972d8c348868 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:05.929468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 806/1614] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 242 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:05.931873 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda2f7ef-1634-478a-a869-cc29f29de050 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:05.932149 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda2f7ef-1634-478a-a869-cc29f29de050 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:05.932415 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda2f7ef-1634-478a-a869-cc29f29de050 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:05.951887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-cda2f7ef-1634-478a-a869-cc29f29de050 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:05.953171 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-cda2f7ef-1634-478a-a869-cc29f29de050 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:05.961132 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 807/1615] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:05.961889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda2f7ef-1634-478a-a869-cc29f29de050 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:05.971199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda2f7ef-1634-478a-a869-cc29f29de050 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:05.978135 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 809/1616] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:05 2022] PUT /identity/v3/projects/2353a6dd9ae24027ae79623008a5d979/users/d675c12bb79f4c95887529e6b26f00ad/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:05.979490 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decf0f97-08a6-4fba-bd2b-d6708982c398 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:05.979903 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decf0f97-08a6-4fba-bd2b-d6708982c398 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:05.980250 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decf0f97-08a6-4fba-bd2b-d6708982c398 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:06.009711 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-decf0f97-08a6-4fba-bd2b-d6708982c398 None None] MFA Rules not processed for user `58f69d8a60a14a2ea87fd840e8b0c916`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:06.010973 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-decf0f97-08a6-4fba-bd2b-d6708982c398 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:06.030585 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda5ee5d-154b-4847-94fd-f22b1ae98d58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:06.031196 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda5ee5d-154b-4847-94fd-f22b1ae98d58 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:06.035190 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda5ee5d-154b-4847-94fd-f22b1ae98d58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:06.035190 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda5ee5d-154b-4847-94fd-f22b1ae98d58 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:06.035190 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda5ee5d-154b-4847-94fd-f22b1ae98d58 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:06.035190 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda5ee5d-154b-4847-94fd-f22b1ae98d58 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:06.088825 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 810/1617] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:05 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:06.090401 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decf0f97-08a6-4fba-bd2b-d6708982c398 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:06.104978 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decf0f97-08a6-4fba-bd2b-d6708982c398 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:06.122926 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 808/1618] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:05 2022] POST /identity/v3/projects => generated 381 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:06.127522 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda5ee5d-154b-4847-94fd-f22b1ae98d58 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:06.137922 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d765e7-5d96-45b1-8123-dc250372a260 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:06.138724 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d765e7-5d96-45b1-8123-dc250372a260 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:06.139146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d765e7-5d96-45b1-8123-dc250372a260 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:06.139441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d765e7-5d96-45b1-8123-dc250372a260 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:06.140093 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d765e7-5d96-45b1-8123-dc250372a260 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:06.143269 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d765e7-5d96-45b1-8123-dc250372a260 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:06.194528 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:06.204568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda5ee5d-154b-4847-94fd-f22b1ae98d58 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:06.219828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2dac037-7d10-41b3-8baf-9aa83d0441a6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:06.220705 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dac037-7d10-41b3-8baf-9aa83d0441a6 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:06.221331 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dac037-7d10-41b3-8baf-9aa83d0441a6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:06.222736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dac037-7d10-41b3-8baf-9aa83d0441a6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:06.269002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 811/1619] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:06 2022] POST /identity/v3/projects => generated 405 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:06.269866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d765e7-5d96-45b1-8123-dc250372a260 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:06.296246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2dac037-7d10-41b3-8baf-9aa83d0441a6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:06.298497 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2dac037-7d10-41b3-8baf-9aa83d0441a6 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:06.311838 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 809/1620] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:06 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:06.313026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2dac037-7d10-41b3-8baf-9aa83d0441a6 service cinder] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:06.364062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d765e7-5d96-45b1-8123-dc250372a260 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:06.370094 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391f00a8-285a-45a7-86d1-9a539fe8cd8c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:06.370726 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391f00a8-285a-45a7-86d1-9a539fe8cd8c service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:06.371026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391f00a8-285a-45a7-86d1-9a539fe8cd8c service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:06.372267 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391f00a8-285a-45a7-86d1-9a539fe8cd8c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:06.432819 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2dac037-7d10-41b3-8baf-9aa83d0441a6 service cinder] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:06.510274 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:06.516812 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391f00a8-285a-45a7-86d1-9a539fe8cd8c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:06.520507 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391f00a8-285a-45a7-86d1-9a539fe8cd8c service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:06.521770 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc1488f-6baf-4832-b7c7-6f56e3ac80d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:06.522453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc1488f-6baf-4832-b7c7-6f56e3ac80d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:06.522793 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc1488f-6baf-4832-b7c7-6f56e3ac80d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:06.523072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc1488f-6baf-4832-b7c7-6f56e3ac80d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:06.524980 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc1488f-6baf-4832-b7c7-6f56e3ac80d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:06.526786 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc1488f-6baf-4832-b7c7-6f56e3ac80d3 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:06.537963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 812/1621] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:06.540374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30579de5-cd60-4847-b669-68c94e1f229b None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:06.540960 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30579de5-cd60-4847-b669-68c94e1f229b None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:06.541496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30579de5-cd60-4847-b669-68c94e1f229b None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:06.549507 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 810/1622] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:06 2022] GET /identity/v3/roles => generated 1420 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:06.550388 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 813/1623] 10.222.0.62 () {54 vars in 961 bytes} [Fri Sep 23 14:02:06 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:02:06.551577 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30579de5-cd60-4847-b669-68c94e1f229b None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:06.553372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc1488f-6baf-4832-b7c7-6f56e3ac80d3 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:06.571812 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30579de5-cd60-4847-b669-68c94e1f229b None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:06.577678 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc1488f-6baf-4832-b7c7-6f56e3ac80d3 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:06.578019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a159d705-f818-4257-96b7-9bac193d00e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:06.578838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a159d705-f818-4257-96b7-9bac193d00e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:06.579204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a159d705-f818-4257-96b7-9bac193d00e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:06.580071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a159d705-f818-4257-96b7-9bac193d00e0 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:06.581277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a159d705-f818-4257-96b7-9bac193d00e0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:06.583899 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a159d705-f818-4257-96b7-9bac193d00e0 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:06.593998 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea186e0f-a515-4ca7-8309-3fb6c2e27b13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:06.594671 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea186e0f-a515-4ca7-8309-3fb6c2e27b13 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:06.594954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea186e0f-a515-4ca7-8309-3fb6c2e27b13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:06.595231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea186e0f-a515-4ca7-8309-3fb6c2e27b13 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:06.595874 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea186e0f-a515-4ca7-8309-3fb6c2e27b13 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:06.600873 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea186e0f-a515-4ca7-8309-3fb6c2e27b13 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:06.714471 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 814/1624] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:06 2022] POST /identity/v3/users => generated 309 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:06.718028 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7742b06-c6dd-4f5e-9601-9d429841b07b None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:06.718028 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7742b06-c6dd-4f5e-9601-9d429841b07b None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:06.718028 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7742b06-c6dd-4f5e-9601-9d429841b07b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:06.744006 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 811/1625] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:06 2022] POST /identity/v3/users => generated 297 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:06.744760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea186e0f-a515-4ca7-8309-3fb6c2e27b13 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:06.768483 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea186e0f-a515-4ca7-8309-3fb6c2e27b13 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:06.781633 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-b7742b06-c6dd-4f5e-9601-9d429841b07b None None] MFA Rules not processed for user `f01ed94dbb384a178ad0a45a6f3d8c17`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:06.781940 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5248cc-ad31-4830-82f9-4ace5247a4a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:06.782600 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5248cc-ad31-4830-82f9-4ace5247a4a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:06.782887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5248cc-ad31-4830-82f9-4ace5247a4a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:06.783165 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5248cc-ad31-4830-82f9-4ace5247a4a6 admin admin] PATH_INFO: `/v3/projects/2353a6dd9ae24027ae79623008a5d979/users/d675c12bb79f4c95887529e6b26f00ad/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:06.784841 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-b7742b06-c6dd-4f5e-9601-9d429841b07b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:06.788492 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5248cc-ad31-4830-82f9-4ace5247a4a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2353a6dd9ae24027ae79623008a5d979, user_id=d675c12bb79f4c95887529e6b26f00ad, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:06.791272 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5248cc-ad31-4830-82f9-4ace5247a4a6 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:06.816630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 812/1626] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:06 2022] PUT /identity/v3/projects/2353a6dd9ae24027ae79623008a5d979/users/d675c12bb79f4c95887529e6b26f00ad/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:06.817485 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5248cc-ad31-4830-82f9-4ace5247a4a6 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:06.938882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 815/1627] 10.222.0.62 () {58 vars in 1058 bytes} [Fri Sep 23 14:02:06 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:06.942376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7742b06-c6dd-4f5e-9601-9d429841b07b None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:06.964412 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5248cc-ad31-4830-82f9-4ace5247a4a6 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:06.966828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7742b06-c6dd-4f5e-9601-9d429841b07b None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:06.976238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e21eae7-d806-4fb3-b169-af4b9361231a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:06.978280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e21eae7-d806-4fb3-b169-af4b9361231a admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:06.979016 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e815fab-49b3-4dbe-89cd-b74b7897f513 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:06.979381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e21eae7-d806-4fb3-b169-af4b9361231a admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:06.979717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e21eae7-d806-4fb3-b169-af4b9361231a admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:06.980388 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e815fab-49b3-4dbe-89cd-b74b7897f513 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:06.981263 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e21eae7-d806-4fb3-b169-af4b9361231a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:06.981576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e815fab-49b3-4dbe-89cd-b74b7897f513 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:06.981914 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e815fab-49b3-4dbe-89cd-b74b7897f513 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:06.983531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e21eae7-d806-4fb3-b169-af4b9361231a admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:06.984274 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e815fab-49b3-4dbe-89cd-b74b7897f513 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:06.985927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e815fab-49b3-4dbe-89cd-b74b7897f513 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:07.007479 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 813/1628] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:06 2022] GET /identity/v3/roles => generated 1420 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:07.009648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11482256-7a28-43d7-819a-fefe75ed634d None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:07.010145 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 816/1629] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:06 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:07.011099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e815fab-49b3-4dbe-89cd-b74b7897f513 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:07.011429 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11482256-7a28-43d7-819a-fefe75ed634d None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:07.011767 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11482256-7a28-43d7-819a-fefe75ed634d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:07.073498 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-11482256-7a28-43d7-819a-fefe75ed634d None None] MFA Rules not processed for user `d675c12bb79f4c95887529e6b26f00ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:07.076737 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-11482256-7a28-43d7-819a-fefe75ed634d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:07.120000 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:07.172927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e815fab-49b3-4dbe-89cd-b74b7897f513 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:07.180100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221e2ced-069c-4324-b70f-43aaac267239 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:07.186275 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221e2ced-069c-4324-b70f-43aaac267239 service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:07.186275 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221e2ced-069c-4324-b70f-43aaac267239 service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:07.186275 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221e2ced-069c-4324-b70f-43aaac267239 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:07.227195 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 814/1630] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:07 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:07.229119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11482256-7a28-43d7-819a-fefe75ed634d None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:07.252944 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:07.254968 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11482256-7a28-43d7-819a-fefe75ed634d None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:07.268506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae81ca47-d74f-46d9-abb2-22dc199edcea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:07.272383 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae81ca47-d74f-46d9-abb2-22dc199edcea admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:07.272383 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae81ca47-d74f-46d9-abb2-22dc199edcea admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:07.272383 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae81ca47-d74f-46d9-abb2-22dc199edcea admin admin] PATH_INFO: `/v3/projects/4dc59805c3224756938a1d714d3e122e/users/2d9ae319549f4328a642443430a1b6ab/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:07.274995 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221e2ced-069c-4324-b70f-43aaac267239 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:07.276610 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221e2ced-069c-4324-b70f-43aaac267239 service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:07.291491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 817/1631] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:07 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 281 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:07.292331 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221e2ced-069c-4324-b70f-43aaac267239 service nova] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:07.299232 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae81ca47-d74f-46d9-abb2-22dc199edcea admin admin] RBAC: Authorizing `identity:create_grant(project_id=4dc59805c3224756938a1d714d3e122e, user_id=2d9ae319549f4328a642443430a1b6ab, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:07.302056 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae81ca47-d74f-46d9-abb2-22dc199edcea admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:07.315052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221e2ced-069c-4324-b70f-43aaac267239 service nova] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:07.321920 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364603c0-3458-47df-8c78-c75cc4c930c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:07.322655 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364603c0-3458-47df-8c78-c75cc4c930c0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:07.322926 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364603c0-3458-47df-8c78-c75cc4c930c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:07.323195 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364603c0-3458-47df-8c78-c75cc4c930c0 admin admin] PATH_INFO: `/v3/projects/addcb88552534c7589ab2d0093d65ab7/users/80aaa35994a54ce5ac2612217c6d7734/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:07.339587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364603c0-3458-47df-8c78-c75cc4c930c0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=addcb88552534c7589ab2d0093d65ab7, user_id=80aaa35994a54ce5ac2612217c6d7734, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:07.342784 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364603c0-3458-47df-8c78-c75cc4c930c0 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:07.393593 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 815/1632] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:07 2022] PUT /identity/v3/projects/4dc59805c3224756938a1d714d3e122e/users/2d9ae319549f4328a642443430a1b6ab/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:07.396540 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae81ca47-d74f-46d9-abb2-22dc199edcea admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:07.420519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 818/1633] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:07 2022] PUT /identity/v3/projects/addcb88552534c7589ab2d0093d65ab7/users/80aaa35994a54ce5ac2612217c6d7734/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:07.421555 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364603c0-3458-47df-8c78-c75cc4c930c0 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:07.513685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae81ca47-d74f-46d9-abb2-22dc199edcea admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:07.654533 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364603c0-3458-47df-8c78-c75cc4c930c0 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:07.654533 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79f8548-1b5a-4c24-82be-7a95958421f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:07.654533 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79f8548-1b5a-4c24-82be-7a95958421f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:07.654533 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79f8548-1b5a-4c24-82be-7a95958421f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:07.657248 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a79f8548-1b5a-4c24-82be-7a95958421f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:07.657248 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79f8548-1b5a-4c24-82be-7a95958421f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:07.657248 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a79f8548-1b5a-4c24-82be-7a95958421f1 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:07.677341 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 819/1634] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:07 2022] GET /identity/v3/roles => generated 1420 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:07.679372 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79f8548-1b5a-4c24-82be-7a95958421f1 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:07.732577 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e122855-f3d8-45fc-98eb-0a07fb86c664 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:07.738538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e122855-f3d8-45fc-98eb-0a07fb86c664 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:07.739081 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e122855-f3d8-45fc-98eb-0a07fb86c664 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:07.739217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e122855-f3d8-45fc-98eb-0a07fb86c664 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:07.744280 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a79f8548-1b5a-4c24-82be-7a95958421f1 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:07.760903 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1db4c3-d772-48a1-853b-489bbb530da7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:07.761533 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1db4c3-d772-48a1-853b-489bbb530da7 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:07.761807 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1db4c3-d772-48a1-853b-489bbb530da7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:07.762068 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1db4c3-d772-48a1-853b-489bbb530da7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:07.769893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1db4c3-d772-48a1-853b-489bbb530da7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:07.787559 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1db4c3-d772-48a1-853b-489bbb530da7 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:07.806546 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:07.807318 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 820/1635] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:07 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:07.809045 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1db4c3-d772-48a1-853b-489bbb530da7 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:08.096208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1db4c3-d772-48a1-853b-489bbb530da7 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:08.096208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f172163-6e19-4d41-810a-498dd2cb2af8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:08.096208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f172163-6e19-4d41-810a-498dd2cb2af8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:08.096208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f172163-6e19-4d41-810a-498dd2cb2af8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:08.097065 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f172163-6e19-4d41-810a-498dd2cb2af8 admin admin] PATH_INFO: `/v3/projects/4dc59805c3224756938a1d714d3e122e/users/2d9ae319549f4328a642443430a1b6ab/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:08.097065 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f172163-6e19-4d41-810a-498dd2cb2af8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4dc59805c3224756938a1d714d3e122e, user_id=2d9ae319549f4328a642443430a1b6ab, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:08.097065 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f172163-6e19-4d41-810a-498dd2cb2af8 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:08.097065 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 821/1636] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:07 2022] PUT /identity/v3/projects/4dc59805c3224756938a1d714d3e122e/users/2d9ae319549f4328a642443430a1b6ab/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:08.097065 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f172163-6e19-4d41-810a-498dd2cb2af8 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:08.098036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e122855-f3d8-45fc-98eb-0a07fb86c664 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:08.098036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e122855-f3d8-45fc-98eb-0a07fb86c664 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:08.098036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 816/1637] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:07 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 591 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:08.098036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040611cf-e571-4c98-b226-14d08d14b685 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:08.098036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040611cf-e571-4c98-b226-14d08d14b685 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:08.098036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040611cf-e571-4c98-b226-14d08d14b685 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:08.104633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-040611cf-e571-4c98-b226-14d08d14b685 None None] MFA Rules not processed for user `2d9ae319549f4328a642443430a1b6ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:08.104633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-040611cf-e571-4c98-b226-14d08d14b685 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:08.106090 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f172163-6e19-4d41-810a-498dd2cb2af8 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:08.120575 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384dadad-83ed-4a0a-8f26-e46ad0198a03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:08.121697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384dadad-83ed-4a0a-8f26-e46ad0198a03 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:08.122198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384dadad-83ed-4a0a-8f26-e46ad0198a03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:08.122715 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384dadad-83ed-4a0a-8f26-e46ad0198a03 admin admin] PATH_INFO: `/v3/projects/addcb88552534c7589ab2d0093d65ab7/users/80aaa35994a54ce5ac2612217c6d7734/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:08.127465 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384dadad-83ed-4a0a-8f26-e46ad0198a03 admin admin] RBAC: Authorizing `identity:create_grant(project_id=addcb88552534c7589ab2d0093d65ab7, user_id=80aaa35994a54ce5ac2612217c6d7734, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:08.134721 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384dadad-83ed-4a0a-8f26-e46ad0198a03 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:08.159489 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 817/1638] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:07 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:08.167185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 822/1639] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:07 2022] PUT /identity/v3/projects/addcb88552534c7589ab2d0093d65ab7/users/80aaa35994a54ce5ac2612217c6d7734/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:08.168731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040611cf-e571-4c98-b226-14d08d14b685 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:08.184954 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b46327-7cbe-4b59-b8aa-d148860a20a1 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:08.185788 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b46327-7cbe-4b59-b8aa-d148860a20a1 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:08.185788 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b46327-7cbe-4b59-b8aa-d148860a20a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:08.285435 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-69b46327-7cbe-4b59-b8aa-d148860a20a1 None None] MFA Rules not processed for user `80aaa35994a54ce5ac2612217c6d7734`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:08.286644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-69b46327-7cbe-4b59-b8aa-d148860a20a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:08.333117 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:08.407920 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040611cf-e571-4c98-b226-14d08d14b685 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:08.456256 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 823/1640] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:08 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 278 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:08.496880 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb580b7-8c9d-44d2-93b4-8ccc2e32264a None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:08.498112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb580b7-8c9d-44d2-93b4-8ccc2e32264a None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:08.500607 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb580b7-8c9d-44d2-93b4-8ccc2e32264a None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:08.502775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 824/1641] 10.222.0.62 () {54 vars in 961 bytes} [Fri Sep 23 14:02:08 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 23 14:02:08.517562 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458e7123-5835-4be6-8c1a-e72039aec57e None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:08.518391 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458e7123-5835-4be6-8c1a-e72039aec57e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:08.518837 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458e7123-5835-4be6-8c1a-e72039aec57e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:08.544903 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef1c890-0359-4c86-a7a1-99d4d2337a2f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:08.545891 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef1c890-0359-4c86-a7a1-99d4d2337a2f service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:08.546279 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef1c890-0359-4c86-a7a1-99d4d2337a2f service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:08.546726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef1c890-0359-4c86-a7a1-99d4d2337a2f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:08.602273 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-458e7123-5835-4be6-8c1a-e72039aec57e None None] MFA Rules not processed for user `f01ed94dbb384a178ad0a45a6f3d8c17`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:08.607589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-458e7123-5835-4be6-8c1a-e72039aec57e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:08.689448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef1c890-0359-4c86-a7a1-99d4d2337a2f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:08.691259 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef1c890-0359-4c86-a7a1-99d4d2337a2f service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:08.705277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 818/1642] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:08 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 537 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:08.777448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef1c890-0359-4c86-a7a1-99d4d2337a2f service cinder] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:08.827262 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef1c890-0359-4c86-a7a1-99d4d2337a2f service cinder] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:08.838127 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eb76f1-b361-4f2f-a669-467abdf03702 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:08.839488 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eb76f1-b361-4f2f-a669-467abdf03702 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:08.839488 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eb76f1-b361-4f2f-a669-467abdf03702 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:08.839488 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eb76f1-b361-4f2f-a669-467abdf03702 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:08.850888 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74eb76f1-b361-4f2f-a669-467abdf03702 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:08.853149 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74eb76f1-b361-4f2f-a669-467abdf03702 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:08.854019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 825/1643] 10.222.0.62 () {58 vars in 1058 bytes} [Fri Sep 23 14:02:08 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 342 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:02:08.861921 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fa59b5-a550-4a1c-86a8-ee80aac618ed None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:08.862211 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fa59b5-a550-4a1c-86a8-ee80aac618ed None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:08.862525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fa59b5-a550-4a1c-86a8-ee80aac618ed None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:08.869468 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 826/1644] 10.222.0.62 () {54 vars in 912 bytes} [Fri Sep 23 14:02:08 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:02:08.873106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0803f7c-936b-4d86-a661-393f359a42af None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:08.873338 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:08.873432 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0803f7c-936b-4d86-a661-393f359a42af None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:08.873648 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0803f7c-936b-4d86-a661-393f359a42af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:08.896018 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 819/1645] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:08 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:08.946755 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c0803f7c-936b-4d86-a661-393f359a42af None None] MFA Rules not processed for user `f01ed94dbb384a178ad0a45a6f3d8c17`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:08.952652 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c0803f7c-936b-4d86-a661-393f359a42af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:08.981210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 827/1646] 10.222.0.62 () {58 vars in 1009 bytes} [Fri Sep 23 14:02:08 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:09.166913 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eb76f1-b361-4f2f-a669-467abdf03702 service cinder] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:09.188348 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eb76f1-b361-4f2f-a669-467abdf03702 service cinder] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:09.202924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770d20a0-1a06-4629-85ba-73f900795d58 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:09.204498 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770d20a0-1a06-4629-85ba-73f900795d58 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:09.208631 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770d20a0-1a06-4629-85ba-73f900795d58 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:09.213620 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770d20a0-1a06-4629-85ba-73f900795d58 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:09.243745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770d20a0-1a06-4629-85ba-73f900795d58 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:09.247316 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770d20a0-1a06-4629-85ba-73f900795d58 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:09.270545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 820/1647] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:09.422979 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70822fa7-9909-45b8-9b25-5e8f39cabd64 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:09.423191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70822fa7-9909-45b8-9b25-5e8f39cabd64 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:09.423445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70822fa7-9909-45b8-9b25-5e8f39cabd64 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:09.424806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 828/1648] 10.222.0.62 () {54 vars in 912 bytes} [Fri Sep 23 14:02:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:02:09.467445 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f430b8b-0907-428d-90ba-a523c697fa70 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:09.468233 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f430b8b-0907-428d-90ba-a523c697fa70 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:09.468756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f430b8b-0907-428d-90ba-a523c697fa70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:09.524759 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70822fa7-9909-45b8-9b25-5e8f39cabd64 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:09.537763 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-7f430b8b-0907-428d-90ba-a523c697fa70 None None] MFA Rules not processed for user `f01ed94dbb384a178ad0a45a6f3d8c17`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:09.540208 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-7f430b8b-0907-428d-90ba-a523c697fa70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:09.544790 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70822fa7-9909-45b8-9b25-5e8f39cabd64 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:09.560782 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d06d3f6-1bd0-4ca0-a01d-d39c9f431f9a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:09.561687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d06d3f6-1bd0-4ca0-a01d-d39c9f431f9a service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:09.562570 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d06d3f6-1bd0-4ca0-a01d-d39c9f431f9a service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:09.562991 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d06d3f6-1bd0-4ca0-a01d-d39c9f431f9a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:09.586503 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 821/1649] 10.222.0.62 () {58 vars in 1009 bytes} [Fri Sep 23 14:02:09 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:09.699187 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d06d3f6-1bd0-4ca0-a01d-d39c9f431f9a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:09.701321 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d06d3f6-1bd0-4ca0-a01d-d39c9f431f9a service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:09.715478 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 829/1650] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:09 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:09.747841 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f430b8b-0907-428d-90ba-a523c697fa70 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:09.767151 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d06d3f6-1bd0-4ca0-a01d-d39c9f431f9a service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:09.772116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d06d3f6-1bd0-4ca0-a01d-d39c9f431f9a service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:09.779866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc192bb-dea5-4c7b-be9b-e05e01512ef2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:09.780680 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc192bb-dea5-4c7b-be9b-e05e01512ef2 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:09.780816 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc192bb-dea5-4c7b-be9b-e05e01512ef2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:09.781009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc192bb-dea5-4c7b-be9b-e05e01512ef2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:09.807590 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc192bb-dea5-4c7b-be9b-e05e01512ef2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:09.809562 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcc192bb-dea5-4c7b-be9b-e05e01512ef2 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:09.827385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 830/1651] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:09.851195 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f430b8b-0907-428d-90ba-a523c697fa70 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:09.869821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b72a7e9-7002-4703-bec0-3b3e0c19cb3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:09.871397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b72a7e9-7002-4703-bec0-3b3e0c19cb3c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:09.873132 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b72a7e9-7002-4703-bec0-3b3e0c19cb3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:09.874137 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b72a7e9-7002-4703-bec0-3b3e0c19cb3c admin admin] PATH_INFO: `/v3/users/bb36a6c7bf3945a39f01605ff5e0af75` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:09.883997 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b72a7e9-7002-4703-bec0-3b3e0c19cb3c admin admin] RBAC: Authorizing `identity:delete_user(user_id=bb36a6c7bf3945a39f01605ff5e0af75)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:09.883997 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: WARNING py.warnings [None req-0b72a7e9-7002-4703-bec0-3b3e0c19cb3c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 23 14:02:09.883997 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: warnings.warn(msg) Sep 23 14:02:09.883997 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: Sep 23 14:02:09.883997 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b72a7e9-7002-4703-bec0-3b3e0c19cb3c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:09.957548 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc192bb-dea5-4c7b-be9b-e05e01512ef2 service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:09.973597 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc192bb-dea5-4c7b-be9b-e05e01512ef2 service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:09.984067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c721e49-ccbb-4a21-8834-745b9accc985 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:09.984067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c721e49-ccbb-4a21-8834-745b9accc985 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:09.984067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c721e49-ccbb-4a21-8834-745b9accc985 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:09.984067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c721e49-ccbb-4a21-8834-745b9accc985 admin admin] PATH_INFO: `/v3/users/58f69d8a60a14a2ea87fd840e8b0c916` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:09.984386 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c721e49-ccbb-4a21-8834-745b9accc985 admin admin] RBAC: Authorizing `identity:delete_user(user_id=58f69d8a60a14a2ea87fd840e8b0c916)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:09.986226 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c721e49-ccbb-4a21-8834-745b9accc985 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:10.003800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:10.202973 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:10.223723 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-0b72a7e9-7002-4703-bec0-3b3e0c19cb3c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bb36a6c7bf3945a39f01605ff5e0af75'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:10.291968 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9c721e49-ccbb-4a21-8834-745b9accc985 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '58f69d8a60a14a2ea87fd840e8b0c916'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:10.356451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-0b72a7e9-7002-4703-bec0-3b3e0c19cb3c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bb36a6c7bf3945a39f01605ff5e0af75'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:10.357690 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-0b72a7e9-7002-4703-bec0-3b3e0c19cb3c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bb36a6c7bf3945a39f01605ff5e0af75'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:10.412392 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-0b72a7e9-7002-4703-bec0-3b3e0c19cb3c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bb36a6c7bf3945a39f01605ff5e0af75'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:10.433961 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-0b72a7e9-7002-4703-bec0-3b3e0c19cb3c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bb36a6c7bf3945a39f01605ff5e0af75'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:10.444258 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 822/1652] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:09 2022] DELETE /identity/v3/users/bb36a6c7bf3945a39f01605ff5e0af75 => generated 0 bytes in 697 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:10.448926 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81c7367-c87d-4d0e-98b8-be83550550ae None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:10.449238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81c7367-c87d-4d0e-98b8-be83550550ae None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:10.450669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81c7367-c87d-4d0e-98b8-be83550550ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:10.475968 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9c721e49-ccbb-4a21-8834-745b9accc985 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '58f69d8a60a14a2ea87fd840e8b0c916'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:10.509626 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-d81c7367-c87d-4d0e-98b8-be83550550ae None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:10.510900 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d81c7367-c87d-4d0e-98b8-be83550550ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:10.583504 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9c721e49-ccbb-4a21-8834-745b9accc985 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '58f69d8a60a14a2ea87fd840e8b0c916'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:10.587958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9c721e49-ccbb-4a21-8834-745b9accc985 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '58f69d8a60a14a2ea87fd840e8b0c916'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:10.617562 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9c721e49-ccbb-4a21-8834-745b9accc985 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '58f69d8a60a14a2ea87fd840e8b0c916'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:10.637019 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 823/1653] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:10.647313 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 831/1654] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:09 2022] DELETE /identity/v3/users/58f69d8a60a14a2ea87fd840e8b0c916 => generated 0 bytes in 691 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:02:10.654014 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cba86d7-4371-4a38-beea-1c7a7e4e87a9 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:10.654306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cba86d7-4371-4a38-beea-1c7a7e4e87a9 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:10.654603 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cba86d7-4371-4a38-beea-1c7a7e4e87a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:10.669905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c721e49-ccbb-4a21-8834-745b9accc985 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:10.671717 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-9c721e49-ccbb-4a21-8834-745b9accc985 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:10.690043 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-9cba86d7-4371-4a38-beea-1c7a7e4e87a9 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:10.695794 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9cba86d7-4371-4a38-beea-1c7a7e4e87a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:10.815897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c721e49-ccbb-4a21-8834-745b9accc985 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:10.821048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938a4325-d9a4-4ef2-ad58-1826f6e496c5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:10.821752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938a4325-d9a4-4ef2-ad58-1826f6e496c5 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:10.822183 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938a4325-d9a4-4ef2-ad58-1826f6e496c5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:10.822322 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938a4325-d9a4-4ef2-ad58-1826f6e496c5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:10.871262 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 824/1655] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:10.939097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938a4325-d9a4-4ef2-ad58-1826f6e496c5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:10.949816 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938a4325-d9a4-4ef2-ad58-1826f6e496c5 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:10.956832 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cba86d7-4371-4a38-beea-1c7a7e4e87a9 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:10.958393 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9cba86d7-4371-4a38-beea-1c7a7e4e87a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:10.971750 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 832/1656] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:10.982965 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cba86d7-4371-4a38-beea-1c7a7e4e87a9 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:10.997936 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd20874d-ac8a-4151-a1fd-a4a796810865 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:10.998595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd20874d-ac8a-4151-a1fd-a4a796810865 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:10.998871 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd20874d-ac8a-4151-a1fd-a4a796810865 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:10.999145 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd20874d-ac8a-4151-a1fd-a4a796810865 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:11.016463 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd20874d-ac8a-4151-a1fd-a4a796810865 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:11.018000 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd20874d-ac8a-4151-a1fd-a4a796810865 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:11.043951 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 825/1657] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:02:12.299449 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938a4325-d9a4-4ef2-ad58-1826f6e496c5 service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:12.301131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-938a4325-d9a4-4ef2-ad58-1826f6e496c5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:12.325393 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938a4325-d9a4-4ef2-ad58-1826f6e496c5 service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:12.343101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0658b332-60b2-4e7b-9c95-03e9169dcdc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:12.343853 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0658b332-60b2-4e7b-9c95-03e9169dcdc3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:12.344199 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0658b332-60b2-4e7b-9c95-03e9169dcdc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:12.344539 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0658b332-60b2-4e7b-9c95-03e9169dcdc3 admin admin] PATH_INFO: `/v3/projects/23b284ca6c134f958ca475088e669dec` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:12.347922 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0658b332-60b2-4e7b-9c95-03e9169dcdc3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=23b284ca6c134f958ca475088e669dec)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:12.349038 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: WARNING py.warnings [None req-0658b332-60b2-4e7b-9c95-03e9169dcdc3 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 23 14:02:12.349038 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: warnings.warn(msg) Sep 23 14:02:12.349038 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: Sep 23 14:02:12.350391 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0658b332-60b2-4e7b-9c95-03e9169dcdc3 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:12.557576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:12.625617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0658b332-60b2-4e7b-9c95-03e9169dcdc3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '23b284ca6c134f958ca475088e669dec', 'request_id': 'req-0658b332-60b2-4e7b-9c95-03e9169dcdc3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:12.708340 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0658b332-60b2-4e7b-9c95-03e9169dcdc3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '23b284ca6c134f958ca475088e669dec', 'request_id': 'req-0658b332-60b2-4e7b-9c95-03e9169dcdc3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:12.720479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0658b332-60b2-4e7b-9c95-03e9169dcdc3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '23b284ca6c134f958ca475088e669dec', 'request_id': 'req-0658b332-60b2-4e7b-9c95-03e9169dcdc3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:12.758372 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0658b332-60b2-4e7b-9c95-03e9169dcdc3 admin admin] The token cache is being invalidate because project 23b284ca6c134f958ca475088e669dec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:12.758650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0658b332-60b2-4e7b-9c95-03e9169dcdc3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:12.760079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0658b332-60b2-4e7b-9c95-03e9169dcdc3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:12.762641 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 833/1658] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:12 2022] DELETE /identity/v3/projects/23b284ca6c134f958ca475088e669dec => generated 0 bytes in 463 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:12.767228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab16ed3b-4968-47ee-8bff-91458c92fc18 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:12.767699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab16ed3b-4968-47ee-8bff-91458c92fc18 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:12.767782 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab16ed3b-4968-47ee-8bff-91458c92fc18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:12.813906 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-ab16ed3b-4968-47ee-8bff-91458c92fc18 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:12.815180 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ab16ed3b-4968-47ee-8bff-91458c92fc18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:12.903594 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:12.957847 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4da95ed-fc60-4c3e-89bc-05b62bb33223 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:12.958436 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4da95ed-fc60-4c3e-89bc-05b62bb33223 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:12.959668 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4da95ed-fc60-4c3e-89bc-05b62bb33223 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:12.967589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 826/1659] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:12.981820 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c4da95ed-fc60-4c3e-89bc-05b62bb33223 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:12.983206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c4da95ed-fc60-4c3e-89bc-05b62bb33223 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:12.997520 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab16ed3b-4968-47ee-8bff-91458c92fc18 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:12.998617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 834/1660] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:13.010143 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab16ed3b-4968-47ee-8bff-91458c92fc18 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:13.034049 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0905ea45-2417-4c6a-accb-01ad4200c68e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:13.034748 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0905ea45-2417-4c6a-accb-01ad4200c68e admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:13.040252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0905ea45-2417-4c6a-accb-01ad4200c68e admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:13.040252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0905ea45-2417-4c6a-accb-01ad4200c68e admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:13.040252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0905ea45-2417-4c6a-accb-01ad4200c68e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:13.040252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0905ea45-2417-4c6a-accb-01ad4200c68e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:13.040252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0905ea45-2417-4c6a-accb-01ad4200c68e admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:13.046456 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4da95ed-fc60-4c3e-89bc-05b62bb33223 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:13.047997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c4da95ed-fc60-4c3e-89bc-05b62bb33223 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:13.068599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4da95ed-fc60-4c3e-89bc-05b62bb33223 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:13.080558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 827/1661] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:02:13.086776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0905ea45-2417-4c6a-accb-01ad4200c68e admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:13.087745 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5f9821-1387-4766-9d4d-91663638b6c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:13.088622 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5f9821-1387-4766-9d4d-91663638b6c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:13.089103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5f9821-1387-4766-9d4d-91663638b6c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:13.090003 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5f9821-1387-4766-9d4d-91663638b6c0 admin admin] PATH_INFO: `/v3/projects/b5ba4efc80634f739469cfdeb8a5e96f` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:13.093460 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5f9821-1387-4766-9d4d-91663638b6c0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b5ba4efc80634f739469cfdeb8a5e96f)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:13.099637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5f9821-1387-4766-9d4d-91663638b6c0 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:13.112289 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0905ea45-2417-4c6a-accb-01ad4200c68e admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:13.127260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b0b49b-1082-4dfa-bdaa-385ae36b312f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:13.128035 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b0b49b-1082-4dfa-bdaa-385ae36b312f admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:13.128416 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b0b49b-1082-4dfa-bdaa-385ae36b312f admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:13.128811 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b0b49b-1082-4dfa-bdaa-385ae36b312f admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:13.129660 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b0b49b-1082-4dfa-bdaa-385ae36b312f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:13.131757 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b0b49b-1082-4dfa-bdaa-385ae36b312f admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:13.270733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 828/1662] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:13 2022] POST /identity/v3/projects => generated 405 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:13.271773 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b0b49b-1082-4dfa-bdaa-385ae36b312f admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:13.308705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fd5f9821-1387-4766-9d4d-91663638b6c0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b5ba4efc80634f739469cfdeb8a5e96f', 'request_id': 'req-fd5f9821-1387-4766-9d4d-91663638b6c0'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:13.440244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fd5f9821-1387-4766-9d4d-91663638b6c0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b5ba4efc80634f739469cfdeb8a5e96f', 'request_id': 'req-fd5f9821-1387-4766-9d4d-91663638b6c0'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:13.458772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fd5f9821-1387-4766-9d4d-91663638b6c0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b5ba4efc80634f739469cfdeb8a5e96f', 'request_id': 'req-fd5f9821-1387-4766-9d4d-91663638b6c0'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:13.479511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b0b49b-1082-4dfa-bdaa-385ae36b312f admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:13.489084 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fd5f9821-1387-4766-9d4d-91663638b6c0 admin admin] The token cache is being invalidate because project b5ba4efc80634f739469cfdeb8a5e96f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:13.490247 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bc2b24-6b0c-4b3d-8ac9-a1c7c1df1684 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:13.492280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bc2b24-6b0c-4b3d-8ac9-a1c7c1df1684 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:13.492716 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bc2b24-6b0c-4b3d-8ac9-a1c7c1df1684 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:13.493368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bc2b24-6b0c-4b3d-8ac9-a1c7c1df1684 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:13.493656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fd5f9821-1387-4766-9d4d-91663638b6c0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:13.494408 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72bc2b24-6b0c-4b3d-8ac9-a1c7c1df1684 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:13.496937 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72bc2b24-6b0c-4b3d-8ac9-a1c7c1df1684 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:13.497408 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fd5f9821-1387-4766-9d4d-91663638b6c0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:13.508992 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 835/1663] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:13 2022] DELETE /identity/v3/projects/b5ba4efc80634f739469cfdeb8a5e96f => generated 0 bytes in 463 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:13.513158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5f9821-1387-4766-9d4d-91663638b6c0 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:13.531752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-fd5f9821-1387-4766-9d4d-91663638b6c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:13.606941 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 829/1664] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:13 2022] POST /identity/v3/users => generated 303 bytes in 335 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:13.612655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bc2b24-6b0c-4b3d-8ac9-a1c7c1df1684 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:13.615823 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-72bc2b24-6b0c-4b3d-8ac9-a1c7c1df1684 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:13.667036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bc2b24-6b0c-4b3d-8ac9-a1c7c1df1684 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:13.695983 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d683911c-ef16-481e-af15-a2b5edce8016 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:13.696828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d683911c-ef16-481e-af15-a2b5edce8016 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:13.697109 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d683911c-ef16-481e-af15-a2b5edce8016 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:13.697461 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d683911c-ef16-481e-af15-a2b5edce8016 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:13.698071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d683911c-ef16-481e-af15-a2b5edce8016 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:13.701841 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d683911c-ef16-481e-af15-a2b5edce8016 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:13.732611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5f9821-1387-4766-9d4d-91663638b6c0 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:13.733751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 830/1665] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:13 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:02:13.737251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d683911c-ef16-481e-af15-a2b5edce8016 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:13.750544 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc433bd6-b2d1-4a56-99e1-4fdf94216e32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:13.751832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc433bd6-b2d1-4a56-99e1-4fdf94216e32 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:13.751832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc433bd6-b2d1-4a56-99e1-4fdf94216e32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:13.752013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc433bd6-b2d1-4a56-99e1-4fdf94216e32 admin admin] PATH_INFO: `/v3/users/2d9ae319549f4328a642443430a1b6ab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:13.753498 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc433bd6-b2d1-4a56-99e1-4fdf94216e32 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2d9ae319549f4328a642443430a1b6ab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:13.755569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc433bd6-b2d1-4a56-99e1-4fdf94216e32 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:13.760873 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d683911c-ef16-481e-af15-a2b5edce8016 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:13.769598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ab0f26-50f4-4152-99b3-cc2b18ec2779 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:13.770314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ab0f26-50f4-4152-99b3-cc2b18ec2779 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:13.770617 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ab0f26-50f4-4152-99b3-cc2b18ec2779 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:13.770889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ab0f26-50f4-4152-99b3-cc2b18ec2779 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:13.771530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ab0f26-50f4-4152-99b3-cc2b18ec2779 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:13.773341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ab0f26-50f4-4152-99b3-cc2b18ec2779 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:13.801610 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 831/1666] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:13 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:02:13.803541 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ab0f26-50f4-4152-99b3-cc2b18ec2779 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:13.841644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ab0f26-50f4-4152-99b3-cc2b18ec2779 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:13.862319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80855a0-4638-4e12-8ded-9e8e6a2847a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:13.863273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80855a0-4638-4e12-8ded-9e8e6a2847a3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:13.863726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80855a0-4638-4e12-8ded-9e8e6a2847a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:13.864784 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80855a0-4638-4e12-8ded-9e8e6a2847a3 admin admin] PATH_INFO: `/v3/projects/04f55434b5cd4bacbb4cd77b47d36015/users/be681f850a2c4f1f93415ea592eba15a/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:13.897234 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80855a0-4638-4e12-8ded-9e8e6a2847a3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=04f55434b5cd4bacbb4cd77b47d36015, user_id=be681f850a2c4f1f93415ea592eba15a, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:13.900678 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80855a0-4638-4e12-8ded-9e8e6a2847a3 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:14.033223 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 832/1667] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:13 2022] PUT /identity/v3/projects/04f55434b5cd4bacbb4cd77b47d36015/users/be681f850a2c4f1f93415ea592eba15a/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:14.041032 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0f9731-045c-4bb0-a908-abf3355948e2 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:14.041519 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0f9731-045c-4bb0-a908-abf3355948e2 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:14.041932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0f9731-045c-4bb0-a908-abf3355948e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:14.084912 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-bb0f9731-045c-4bb0-a908-abf3355948e2 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:14.093300 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-bb0f9731-045c-4bb0-a908-abf3355948e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:14.101931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fc433bd6-b2d1-4a56-99e1-4fdf94216e32 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2d9ae319549f4328a642443430a1b6ab'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:14.193089 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fc433bd6-b2d1-4a56-99e1-4fdf94216e32 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2d9ae319549f4328a642443430a1b6ab'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:14.236462 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 833/1668] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:14.238551 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fc433bd6-b2d1-4a56-99e1-4fdf94216e32 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2d9ae319549f4328a642443430a1b6ab'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:14.250148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fc433bd6-b2d1-4a56-99e1-4fdf94216e32 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2d9ae319549f4328a642443430a1b6ab'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:14.264324 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fc433bd6-b2d1-4a56-99e1-4fdf94216e32 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2d9ae319549f4328a642443430a1b6ab'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:14.266726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb0f9731-045c-4bb0-a908-abf3355948e2 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:14.269356 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-bb0f9731-045c-4bb0-a908-abf3355948e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:14.278366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 836/1669] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:13 2022] DELETE /identity/v3/users/2d9ae319549f4328a642443430a1b6ab => generated 0 bytes in 766 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:14.368126 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb0f9731-045c-4bb0-a908-abf3355948e2 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:14.476542 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb78f43-811b-43ca-b610-cd126b569dbf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:14.477474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb78f43-811b-43ca-b610-cd126b569dbf service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:14.477776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb78f43-811b-43ca-b610-cd126b569dbf service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:14.478021 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb78f43-811b-43ca-b610-cd126b569dbf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:14.484541 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-abb78f43-811b-43ca-b610-cd126b569dbf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:14.610865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb78f43-811b-43ca-b610-cd126b569dbf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:14.612322 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb78f43-811b-43ca-b610-cd126b569dbf service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:14.637330 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 834/1670] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:02:14.768845 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:15.090628 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:15.553178 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf668a89-9315-46e9-9a12-2768c32f2bfe None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:15.553560 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf668a89-9315-46e9-9a12-2768c32f2bfe None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:15.553932 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf668a89-9315-46e9-9a12-2768c32f2bfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:15.590746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-cf668a89-9315-46e9-9a12-2768c32f2bfe None None] MFA Rules not processed for user `28b64a35d7b44519a1d93919f2e23083`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:15.592030 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-cf668a89-9315-46e9-9a12-2768c32f2bfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:15.697341 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 837/1671] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:15 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:15.707939 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388ec6b4-a0ef-4781-95d1-809038af40f9 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:15.708212 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388ec6b4-a0ef-4781-95d1-809038af40f9 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:15.714001 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-388ec6b4-a0ef-4781-95d1-809038af40f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:15.739491 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-388ec6b4-a0ef-4781-95d1-809038af40f9 None None] MFA Rules not processed for user `28b64a35d7b44519a1d93919f2e23083`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:15.741013 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-388ec6b4-a0ef-4781-95d1-809038af40f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:15.750089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 835/1672] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:15 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:15.763101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf668a89-9315-46e9-9a12-2768c32f2bfe None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:15.766085 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-cf668a89-9315-46e9-9a12-2768c32f2bfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:15.880963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf668a89-9315-46e9-9a12-2768c32f2bfe None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:15.893597 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388ec6b4-a0ef-4781-95d1-809038af40f9 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:15.899932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-388ec6b4-a0ef-4781-95d1-809038af40f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:15.903908 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c30abe3-4bd9-4456-9cb5-df5f51ad6ac7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:15.907494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c30abe3-4bd9-4456-9cb5-df5f51ad6ac7 service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:15.913767 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c30abe3-4bd9-4456-9cb5-df5f51ad6ac7 service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:15.917609 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c30abe3-4bd9-4456-9cb5-df5f51ad6ac7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:15.932612 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-388ec6b4-a0ef-4781-95d1-809038af40f9 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:15.935150 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c30abe3-4bd9-4456-9cb5-df5f51ad6ac7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:15.938118 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c30abe3-4bd9-4456-9cb5-df5f51ad6ac7 service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:15.947327 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b26069-4ae2-4589-8e71-718fd35f92f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:15.948179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b26069-4ae2-4589-8e71-718fd35f92f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:15.948620 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b26069-4ae2-4589-8e71-718fd35f92f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:15.949053 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b26069-4ae2-4589-8e71-718fd35f92f8 admin admin] PATH_INFO: `/v3/projects/4dc59805c3224756938a1d714d3e122e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:15.952149 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b26069-4ae2-4589-8e71-718fd35f92f8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4dc59805c3224756938a1d714d3e122e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:15.955264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b26069-4ae2-4589-8e71-718fd35f92f8 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:15.966122 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 838/1673] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:15 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:02:16.049119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c30abe3-4bd9-4456-9cb5-df5f51ad6ac7 service nova] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:16.050765 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-5c30abe3-4bd9-4456-9cb5-df5f51ad6ac7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:16.165536 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-86b26069-4ae2-4589-8e71-718fd35f92f8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4dc59805c3224756938a1d714d3e122e', 'request_id': 'req-86b26069-4ae2-4589-8e71-718fd35f92f8'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:16.251575 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c30abe3-4bd9-4456-9cb5-df5f51ad6ac7 service nova] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:16.272086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-86b26069-4ae2-4589-8e71-718fd35f92f8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4dc59805c3224756938a1d714d3e122e', 'request_id': 'req-86b26069-4ae2-4589-8e71-718fd35f92f8'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:16.279990 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-86b26069-4ae2-4589-8e71-718fd35f92f8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4dc59805c3224756938a1d714d3e122e', 'request_id': 'req-86b26069-4ae2-4589-8e71-718fd35f92f8'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:16.315902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-86b26069-4ae2-4589-8e71-718fd35f92f8 admin admin] The token cache is being invalidate because project 4dc59805c3224756938a1d714d3e122e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:16.316205 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-86b26069-4ae2-4589-8e71-718fd35f92f8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:16.321388 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-86b26069-4ae2-4589-8e71-718fd35f92f8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:16.329468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 836/1674] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:15 2022] DELETE /identity/v3/projects/4dc59805c3224756938a1d714d3e122e => generated 0 bytes in 437 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:16.336542 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f623518a-e270-409a-a26e-1feecda188f7 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:16.336542 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f623518a-e270-409a-a26e-1feecda188f7 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:16.342752 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f623518a-e270-409a-a26e-1feecda188f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:16.405626 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-f623518a-e270-409a-a26e-1feecda188f7 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:16.406564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f623518a-e270-409a-a26e-1feecda188f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:16.431735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d353a3-9309-4f5b-a301-cce56361f5db service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:16.432371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d353a3-9309-4f5b-a301-cce56361f5db service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:16.432650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d353a3-9309-4f5b-a301-cce56361f5db service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:16.432922 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d353a3-9309-4f5b-a301-cce56361f5db service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:16.441093 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-64d353a3-9309-4f5b-a301-cce56361f5db service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:16.530664 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:16.577440 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d353a3-9309-4f5b-a301-cce56361f5db service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:16.579197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d353a3-9309-4f5b-a301-cce56361f5db service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:16.600143 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 839/1675] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:16 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 551 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:02:16.666958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:DELETE=4,keystone:SELECT=95,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:16.687618 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f47c9b-508d-4c38-aec4-35944d56d042 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:16.688205 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f47c9b-508d-4c38-aec4-35944d56d042 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:16.688651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f47c9b-508d-4c38-aec4-35944d56d042 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:16.688928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 837/1676] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 354 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:16.729394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-91f47c9b-508d-4c38-aec4-35944d56d042 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:16.733818 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-91f47c9b-508d-4c38-aec4-35944d56d042 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:16.751447 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 840/1677] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:02:16.758957 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f623518a-e270-409a-a26e-1feecda188f7 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:16.788861 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f623518a-e270-409a-a26e-1feecda188f7 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:16.795176 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74298d1b-8320-4743-975e-acdce6a8fb7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:16.796106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74298d1b-8320-4743-975e-acdce6a8fb7a admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:16.797064 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74298d1b-8320-4743-975e-acdce6a8fb7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:16.798009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74298d1b-8320-4743-975e-acdce6a8fb7a admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:16.798898 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74298d1b-8320-4743-975e-acdce6a8fb7a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:16.800511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74298d1b-8320-4743-975e-acdce6a8fb7a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:16.802526 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74298d1b-8320-4743-975e-acdce6a8fb7a admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:16.826846 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 838/1678] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:02:16.827654 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f47c9b-508d-4c38-aec4-35944d56d042 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:16.838178 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f47c9b-508d-4c38-aec4-35944d56d042 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:16.844524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9f2f17-fd61-4019-b78b-a20924b56f58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:16.845405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9f2f17-fd61-4019-b78b-a20924b56f58 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:16.845872 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9f2f17-fd61-4019-b78b-a20924b56f58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:16.846354 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9f2f17-fd61-4019-b78b-a20924b56f58 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:16.847194 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9f2f17-fd61-4019-b78b-a20924b56f58 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:16.849910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9f2f17-fd61-4019-b78b-a20924b56f58 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:16.967287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74298d1b-8320-4743-975e-acdce6a8fb7a admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:16.973366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 841/1679] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:16 2022] POST /identity/v3/projects => generated 359 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:02:16.984470 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74298d1b-8320-4743-975e-acdce6a8fb7a admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:17.162852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626d5fa6-7272-4b37-a666-317eabe2a12b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:17.165462 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626d5fa6-7272-4b37-a666-317eabe2a12b admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:17.165679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626d5fa6-7272-4b37-a666-317eabe2a12b admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:17.166204 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626d5fa6-7272-4b37-a666-317eabe2a12b admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:17.166818 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626d5fa6-7272-4b37-a666-317eabe2a12b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:17.168827 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626d5fa6-7272-4b37-a666-317eabe2a12b admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:17.325218 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 839/1680] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:16 2022] POST /identity/v3/users => generated 280 bytes in 358 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:17.329664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9f2f17-fd61-4019-b78b-a20924b56f58 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:17.345645 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9f2f17-fd61-4019-b78b-a20924b56f58 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:17.361815 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316b86a1-f4ac-405a-a683-d300c0794208 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:17.362474 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316b86a1-f4ac-405a-a683-d300c0794208 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:17.362762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316b86a1-f4ac-405a-a683-d300c0794208 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:17.363037 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316b86a1-f4ac-405a-a683-d300c0794208 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:17.363696 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316b86a1-f4ac-405a-a683-d300c0794208 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:17.365291 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-316b86a1-f4ac-405a-a683-d300c0794208 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:17.390206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626d5fa6-7272-4b37-a666-317eabe2a12b admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:17.393241 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 842/1681] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:17 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:17.399891 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626d5fa6-7272-4b37-a666-317eabe2a12b admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:17.409723 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495462e7-a5c9-4d18-a6ab-2c1e35cd2d29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:17.410731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495462e7-a5c9-4d18-a6ab-2c1e35cd2d29 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:17.411180 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495462e7-a5c9-4d18-a6ab-2c1e35cd2d29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:17.411664 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495462e7-a5c9-4d18-a6ab-2c1e35cd2d29 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:17.412460 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495462e7-a5c9-4d18-a6ab-2c1e35cd2d29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:17.421111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495462e7-a5c9-4d18-a6ab-2c1e35cd2d29 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:17.457909 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316b86a1-f4ac-405a-a683-d300c0794208 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:17.458880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 840/1682] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:17 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:17.478102 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316b86a1-f4ac-405a-a683-d300c0794208 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:17.523802 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ae58c5-bc78-4bd7-ac46-b8bec38510fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:17.526453 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ae58c5-bc78-4bd7-ac46-b8bec38510fa admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:17.526798 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ae58c5-bc78-4bd7-ac46-b8bec38510fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:17.527079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ae58c5-bc78-4bd7-ac46-b8bec38510fa admin admin] PATH_INFO: `/v3/projects/a011b7f9815648bbbd033072449a4b5f/users/c16d23b1175f46e1b69a844ee82c5295/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:17.575876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ae58c5-bc78-4bd7-ac46-b8bec38510fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=a011b7f9815648bbbd033072449a4b5f, user_id=c16d23b1175f46e1b69a844ee82c5295, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:17.593819 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1ae58c5-bc78-4bd7-ac46-b8bec38510fa admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:17.691728 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 843/1683] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:17 2022] PUT /identity/v3/projects/a011b7f9815648bbbd033072449a4b5f/users/c16d23b1175f46e1b69a844ee82c5295/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:02:17.704125 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef3f86a-9c3b-45aa-99ca-fde150f067fc None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:17.704125 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef3f86a-9c3b-45aa-99ca-fde150f067fc None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:17.704125 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef3f86a-9c3b-45aa-99ca-fde150f067fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:17.777189 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-eef3f86a-9c3b-45aa-99ca-fde150f067fc None None] MFA Rules not processed for user `c16d23b1175f46e1b69a844ee82c5295`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:17.777631 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-eef3f86a-9c3b-45aa-99ca-fde150f067fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:17.826292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9902b8-9491-4017-8ddd-bbaa9f212af9 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:17.826604 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9902b8-9491-4017-8ddd-bbaa9f212af9 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:17.827035 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9902b8-9491-4017-8ddd-bbaa9f212af9 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:17.828774 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 844/1684] 10.222.0.62 () {54 vars in 912 bytes} [Fri Sep 23 14:02:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:02:17.843003 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9af1ac-9a1c-4fdd-9359-a01873d7d610 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:17.843515 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9af1ac-9a1c-4fdd-9359-a01873d7d610 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:17.843956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9af1ac-9a1c-4fdd-9359-a01873d7d610 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:17.904461 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:INSERT=4,keystone:DELETE=3,keystone:SELECT=93 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:17.956787 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-1b9af1ac-9a1c-4fdd-9359-a01873d7d610 None None] MFA Rules not processed for user `580a35627c234a25aa04f5d376177ca5`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:17.958735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-1b9af1ac-9a1c-4fdd-9359-a01873d7d610 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:18.014833 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 841/1685] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:17 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:18.015826 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef3f86a-9c3b-45aa-99ca-fde150f067fc None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:18.107444 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 845/1686] 10.222.0.62 () {58 vars in 1009 bytes} [Fri Sep 23 14:02:17 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 266 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:02:18.142378 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9af1ac-9a1c-4fdd-9359-a01873d7d610 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:18.143967 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-1b9af1ac-9a1c-4fdd-9359-a01873d7d610 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:18.161846 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef3f86a-9c3b-45aa-99ca-fde150f067fc None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:18.165729 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9af1ac-9a1c-4fdd-9359-a01873d7d610 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:18.170155 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d356927f-d1b4-4c9f-ac78-fe428b0c552b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:18.171357 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d356927f-d1b4-4c9f-ac78-fe428b0c552b admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:18.171779 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d356927f-d1b4-4c9f-ac78-fe428b0c552b admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:18.172277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d356927f-d1b4-4c9f-ac78-fe428b0c552b admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:18.173532 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d356927f-d1b4-4c9f-ac78-fe428b0c552b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:18.175925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d356927f-d1b4-4c9f-ac78-fe428b0c552b admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:18.177474 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d458bf-6fe8-4b74-8727-e694d6ec377e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:18.178199 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d458bf-6fe8-4b74-8727-e694d6ec377e service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:18.179034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d458bf-6fe8-4b74-8727-e694d6ec377e service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:18.179313 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d458bf-6fe8-4b74-8727-e694d6ec377e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:18.199589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d458bf-6fe8-4b74-8727-e694d6ec377e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:18.201110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d458bf-6fe8-4b74-8727-e694d6ec377e service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:18.231433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:18.261147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 846/1687] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:18 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:02:18.367901 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 842/1688] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:18 2022] POST /identity/v3/projects => generated 363 bytes in 352 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:18.368885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d458bf-6fe8-4b74-8727-e694d6ec377e service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:18.387068 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d458bf-6fe8-4b74-8727-e694d6ec377e service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:18.513738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7e9d48-a6b8-4ddb-ad78-0c62259b25d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:18.514417 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7e9d48-a6b8-4ddb-ad78-0c62259b25d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:18.514666 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7e9d48-a6b8-4ddb-ad78-0c62259b25d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:18.514931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7e9d48-a6b8-4ddb-ad78-0c62259b25d7 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:18.515571 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7e9d48-a6b8-4ddb-ad78-0c62259b25d7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:18.519913 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7e9d48-a6b8-4ddb-ad78-0c62259b25d7 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:18.633230 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 847/1689] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:18 2022] POST /identity/v3/users => generated 288 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:02:18.634473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d356927f-d1b4-4c9f-ac78-fe428b0c552b admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:18.653855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d356927f-d1b4-4c9f-ac78-fe428b0c552b admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:18.662663 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30f744a-0399-44d9-b718-3c53aed02f2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:18.663918 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30f744a-0399-44d9-b718-3c53aed02f2c admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:18.664217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30f744a-0399-44d9-b718-3c53aed02f2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:18.664493 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30f744a-0399-44d9-b718-3c53aed02f2c admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:18.665171 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30f744a-0399-44d9-b718-3c53aed02f2c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:18.670935 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e30f744a-0399-44d9-b718-3c53aed02f2c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:18.702512 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 843/1690] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:18 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:18.703724 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7e9d48-a6b8-4ddb-ad78-0c62259b25d7 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:18.718395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7e9d48-a6b8-4ddb-ad78-0c62259b25d7 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:18.740784 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b710038c-0c57-4d84-a4fa-d90c052086a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:18.745653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b710038c-0c57-4d84-a4fa-d90c052086a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:18.745653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b710038c-0c57-4d84-a4fa-d90c052086a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:18.745653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b710038c-0c57-4d84-a4fa-d90c052086a7 admin admin] PATH_INFO: `/v3/projects/e49858138ae041228ad1176c68843484/users/b8d762b3372341ba86f26bdf30cb3bb3/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:18.789110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b710038c-0c57-4d84-a4fa-d90c052086a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e49858138ae041228ad1176c68843484, user_id=b8d762b3372341ba86f26bdf30cb3bb3, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:18.795667 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b710038c-0c57-4d84-a4fa-d90c052086a7 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:18.946749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 848/1691] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:18 2022] PUT /identity/v3/projects/e49858138ae041228ad1176c68843484/users/b8d762b3372341ba86f26bdf30cb3bb3/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:18.948060 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30f744a-0399-44d9-b718-3c53aed02f2c admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:18.962753 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30f744a-0399-44d9-b718-3c53aed02f2c admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:19.106407 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c0ea9d-6c42-4d05-bcee-f3ec60ddfa42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:19.109758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c0ea9d-6c42-4d05-bcee-f3ec60ddfa42 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:19.109758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c0ea9d-6c42-4d05-bcee-f3ec60ddfa42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:19.109758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c0ea9d-6c42-4d05-bcee-f3ec60ddfa42 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:19.109758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c0ea9d-6c42-4d05-bcee-f3ec60ddfa42 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:19.110358 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c0ea9d-6c42-4d05-bcee-f3ec60ddfa42 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:19.151932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 844/1692] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:18 2022] GET /identity/v3/roles => generated 1420 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:19.156154 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b710038c-0c57-4d84-a4fa-d90c052086a7 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:19.171270 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b710038c-0c57-4d84-a4fa-d90c052086a7 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:19.197679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85cdbc6-f06f-497a-9678-dc365db9a097 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:19.197813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85cdbc6-f06f-497a-9678-dc365db9a097 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:19.198043 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85cdbc6-f06f-497a-9678-dc365db9a097 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:19.198347 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85cdbc6-f06f-497a-9678-dc365db9a097 admin admin] PATH_INFO: `/v3/projects/e49858138ae041228ad1176c68843484/users/b8d762b3372341ba86f26bdf30cb3bb3/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:19.214823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85cdbc6-f06f-497a-9678-dc365db9a097 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e49858138ae041228ad1176c68843484, user_id=b8d762b3372341ba86f26bdf30cb3bb3, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:19.218875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85cdbc6-f06f-497a-9678-dc365db9a097 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:19.274439 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 849/1693] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:19 2022] PUT /identity/v3/projects/e49858138ae041228ad1176c68843484/users/b8d762b3372341ba86f26bdf30cb3bb3/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:19.290158 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1d4d9a-f701-4f42-9682-8cf0c9055d6b None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:19.292808 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1d4d9a-f701-4f42-9682-8cf0c9055d6b None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:19.293351 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1d4d9a-f701-4f42-9682-8cf0c9055d6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:19.364960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-3c1d4d9a-f701-4f42-9682-8cf0c9055d6b None None] MFA Rules not processed for user `b8d762b3372341ba86f26bdf30cb3bb3`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:19.364960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3c1d4d9a-f701-4f42-9682-8cf0c9055d6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:19.523676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:19.597715 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 845/1694] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:19 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 311 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:19.605118 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa267c21-2e30-4c3f-a759-62cf2bce2f8f None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:19.607235 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1d4d9a-f701-4f42-9682-8cf0c9055d6b None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:19.609736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3c1d4d9a-f701-4f42-9682-8cf0c9055d6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:19.619393 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa267c21-2e30-4c3f-a759-62cf2bce2f8f None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:19.619973 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa267c21-2e30-4c3f-a759-62cf2bce2f8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:19.714464 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-fa267c21-2e30-4c3f-a759-62cf2bce2f8f None None] MFA Rules not processed for user `be681f850a2c4f1f93415ea592eba15a`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:19.715776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-fa267c21-2e30-4c3f-a759-62cf2bce2f8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:19.758453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1d4d9a-f701-4f42-9682-8cf0c9055d6b None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:19.773396 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-463e0b92-6700-452e-891a-26634091079f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:19.774091 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463e0b92-6700-452e-891a-26634091079f service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:19.774388 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463e0b92-6700-452e-891a-26634091079f service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:19.774661 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-463e0b92-6700-452e-891a-26634091079f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:19.811471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463e0b92-6700-452e-891a-26634091079f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:19.814308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-463e0b92-6700-452e-891a-26634091079f service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:19.848868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 846/1695] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:19 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 242 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:19.853241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da4119a-2cd5-46cb-8a34-ff2596c3eee1 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:19.853827 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da4119a-2cd5-46cb-8a34-ff2596c3eee1 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:19.854239 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da4119a-2cd5-46cb-8a34-ff2596c3eee1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:19.930204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 850/1696] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:19 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 330 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:19.932440 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-3da4119a-2cd5-46cb-8a34-ff2596c3eee1 None None] MFA Rules not processed for user `0db30f990aae49fabf9b8b19cd019de9`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:19.933645 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa267c21-2e30-4c3f-a759-62cf2bce2f8f None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:19.934607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3da4119a-2cd5-46cb-8a34-ff2596c3eee1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:19.937074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-fa267c21-2e30-4c3f-a759-62cf2bce2f8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:19.950376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:20.168690 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa267c21-2e30-4c3f-a759-62cf2bce2f8f None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:20.176158 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 847/1697] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:19 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 326 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:20.177142 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fef77c-7889-4969-b955-1cea15d8d409 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce249d4a99c34891b6ba941f57a14dd4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:20.178121 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fef77c-7889-4969-b955-1cea15d8d409 service placement] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:20.179071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fef77c-7889-4969-b955-1cea15d8d409 service placement] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:20.179478 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fef77c-7889-4969-b955-1cea15d8d409 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:20.180210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da4119a-2cd5-46cb-8a34-ff2596c3eee1 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:20.187322 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3da4119a-2cd5-46cb-8a34-ff2596c3eee1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:20.198933 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-07fef77c-7889-4969-b955-1cea15d8d409 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:20.222097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07fef77c-7889-4969-b955-1cea15d8d409 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:20.224722 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07fef77c-7889-4969-b955-1cea15d8d409 service placement] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:20.262943 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 851/1698] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:19 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:20.264263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a93f4c-3890-42ec-960e-252eeaa58d0e None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:20.264547 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a93f4c-3890-42ec-960e-252eeaa58d0e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:20.264826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a93f4c-3890-42ec-960e-252eeaa58d0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:20.315585 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-d5a93f4c-3890-42ec-960e-252eeaa58d0e None None] MFA Rules not processed for user `28b64a35d7b44519a1d93919f2e23083`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:20.324022 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d5a93f4c-3890-42ec-960e-252eeaa58d0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:20.405484 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da4119a-2cd5-46cb-8a34-ff2596c3eee1 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:20.429763 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195d0219-44e2-494f-8720-96d33893a99f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:20.432500 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195d0219-44e2-494f-8720-96d33893a99f admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:20.432500 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195d0219-44e2-494f-8720-96d33893a99f admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:20.434966 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195d0219-44e2-494f-8720-96d33893a99f admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:20.436311 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195d0219-44e2-494f-8720-96d33893a99f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:20.442084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195d0219-44e2-494f-8720-96d33893a99f admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:20.469675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:20.548874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 852/1699] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:20 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 285 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:20.549944 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a93f4c-3890-42ec-960e-252eeaa58d0e None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:20.552263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d5a93f4c-3890-42ec-960e-252eeaa58d0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:20.638926 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 848/1700] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:20 2022] POST /identity/v3/projects => generated 405 bytes in 462 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:20.640355 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be65f0e1-29bc-4a01-bb28-d432c1768912 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:20.640743 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be65f0e1-29bc-4a01-bb28-d432c1768912 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:20.641170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be65f0e1-29bc-4a01-bb28-d432c1768912 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:20.743517 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-be65f0e1-29bc-4a01-bb28-d432c1768912 None None] MFA Rules not processed for user `88a8db8d7d494c62b65e427278ef9a2a`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:20.745358 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-be65f0e1-29bc-4a01-bb28-d432c1768912 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:20.810320 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a93f4c-3890-42ec-960e-252eeaa58d0e None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:20.888989 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:20.953435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 849/1701] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:20 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 314 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:20.957637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be65f0e1-29bc-4a01-bb28-d432c1768912 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:20.998062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c683b9-18a5-4b4d-b684-5bf1788a48e2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:21.001582 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c683b9-18a5-4b4d-b684-5bf1788a48e2 service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:21.002013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c683b9-18a5-4b4d-b684-5bf1788a48e2 service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:21.002292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c683b9-18a5-4b4d-b684-5bf1788a48e2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:21.105970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be65f0e1-29bc-4a01-bb28-d432c1768912 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:21.112208 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4236727-c447-4d79-8260-9c97f450b3cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:21.112831 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4236727-c447-4d79-8260-9c97f450b3cd admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:21.113106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4236727-c447-4d79-8260-9c97f450b3cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:21.113373 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4236727-c447-4d79-8260-9c97f450b3cd admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:21.114002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4236727-c447-4d79-8260-9c97f450b3cd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:21.116714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4236727-c447-4d79-8260-9c97f450b3cd admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:21.136609 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c683b9-18a5-4b4d-b684-5bf1788a48e2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:21.138634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c683b9-18a5-4b4d-b684-5bf1788a48e2 service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:21.169629 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 853/1702] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:20 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 620 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:21.188882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446f3142-407a-4955-892c-0e50a511c2d3 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:21.190812 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446f3142-407a-4955-892c-0e50a511c2d3 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:21.191391 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446f3142-407a-4955-892c-0e50a511c2d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:21.220871 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-446f3142-407a-4955-892c-0e50a511c2d3 None None] MFA Rules not processed for user `88a8db8d7d494c62b65e427278ef9a2a`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:21.222778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-446f3142-407a-4955-892c-0e50a511c2d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:21.227129 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 850/1703] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:20 2022] POST /identity/v3/users => generated 309 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:21.227129 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1112830-3aa3-4f05-ab27-03d3ea615ac6 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:21.227129 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1112830-3aa3-4f05-ab27-03d3ea615ac6 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:21.227129 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1112830-3aa3-4f05-ab27-03d3ea615ac6 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:21.227435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 851/1704] 10.222.0.62 () {54 vars in 912 bytes} [Fri Sep 23 14:02:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:02:21.230291 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1112830-3aa3-4f05-ab27-03d3ea615ac6 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:21.246444 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 854/1705] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:21 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:21.247251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fe0f9f-c601-4ac2-b567-4e21752ebdea None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:21.247650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fe0f9f-c601-4ac2-b567-4e21752ebdea None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:21.248032 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fe0f9f-c601-4ac2-b567-4e21752ebdea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:21.268382 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1112830-3aa3-4f05-ab27-03d3ea615ac6 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:21.276686 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-f4fe0f9f-c601-4ac2-b567-4e21752ebdea None None] MFA Rules not processed for user `580a35627c234a25aa04f5d376177ca5`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:21.278174 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f4fe0f9f-c601-4ac2-b567-4e21752ebdea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:21.284277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0f965d-1e73-49b9-b5bb-7f8e0951ef87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:21.289597 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0f965d-1e73-49b9-b5bb-7f8e0951ef87 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:21.289886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0f965d-1e73-49b9-b5bb-7f8e0951ef87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:21.290338 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0f965d-1e73-49b9-b5bb-7f8e0951ef87 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:21.291039 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0f965d-1e73-49b9-b5bb-7f8e0951ef87 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:21.300017 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b0f965d-1e73-49b9-b5bb-7f8e0951ef87 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:21.310977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 852/1706] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:21 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:21.315211 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0f965d-1e73-49b9-b5bb-7f8e0951ef87 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:21.334302 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0f965d-1e73-49b9-b5bb-7f8e0951ef87 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:21.349117 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbef8f3-b268-4278-a81a-8433c30f770f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:21.350713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbef8f3-b268-4278-a81a-8433c30f770f admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:21.351857 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbef8f3-b268-4278-a81a-8433c30f770f admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:21.352597 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbef8f3-b268-4278-a81a-8433c30f770f admin admin] PATH_INFO: `/v3/projects/da4abb67ad4d4464b74e5ccbbd09b426/users/3432bae1c2ed4f6fa40333efc65f971c/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:21.370970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbef8f3-b268-4278-a81a-8433c30f770f admin admin] RBAC: Authorizing `identity:create_grant(project_id=da4abb67ad4d4464b74e5ccbbd09b426, user_id=3432bae1c2ed4f6fa40333efc65f971c, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:21.374444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbef8f3-b268-4278-a81a-8433c30f770f admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:21.405461 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 855/1707] 10.222.0.62 () {58 vars in 1009 bytes} [Fri Sep 23 14:02:21 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:21.406358 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4fe0f9f-c601-4ac2-b567-4e21752ebdea None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:21.420293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4fe0f9f-c601-4ac2-b567-4e21752ebdea None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:21.426065 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6191f963-e3e8-44ba-9ddb-7c1c8b526f5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:21.426742 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6191f963-e3e8-44ba-9ddb-7c1c8b526f5b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:21.427021 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6191f963-e3e8-44ba-9ddb-7c1c8b526f5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:21.427289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6191f963-e3e8-44ba-9ddb-7c1c8b526f5b admin admin] PATH_INFO: `/v3/users/c16d23b1175f46e1b69a844ee82c5295` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:21.429128 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6191f963-e3e8-44ba-9ddb-7c1c8b526f5b admin admin] RBAC: Authorizing `identity:delete_user(user_id=c16d23b1175f46e1b69a844ee82c5295)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:21.431650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6191f963-e3e8-44ba-9ddb-7c1c8b526f5b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:21.484715 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 853/1708] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:21 2022] PUT /identity/v3/projects/da4abb67ad4d4464b74e5ccbbd09b426/users/3432bae1c2ed4f6fa40333efc65f971c/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:02:21.487767 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbef8f3-b268-4278-a81a-8433c30f770f admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:21.541384 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:21.650135 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbef8f3-b268-4278-a81a-8433c30f770f admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:21.670535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6191f963-e3e8-44ba-9ddb-7c1c8b526f5b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c16d23b1175f46e1b69a844ee82c5295'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:21.675223 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f098df-f435-48c9-a831-0e450ef6ef03 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:21.676060 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f098df-f435-48c9-a831-0e450ef6ef03 service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:21.677337 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f098df-f435-48c9-a831-0e450ef6ef03 service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:21.679572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f098df-f435-48c9-a831-0e450ef6ef03 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:21.776086 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6191f963-e3e8-44ba-9ddb-7c1c8b526f5b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c16d23b1175f46e1b69a844ee82c5295'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:21.776445 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f098df-f435-48c9-a831-0e450ef6ef03 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:21.779201 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f098df-f435-48c9-a831-0e450ef6ef03 service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:21.802839 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 854/1709] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:21 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:21.804416 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f098df-f435-48c9-a831-0e450ef6ef03 service nova] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:21.815246 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:21.818419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6191f963-e3e8-44ba-9ddb-7c1c8b526f5b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c16d23b1175f46e1b69a844ee82c5295'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:21.821501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6191f963-e3e8-44ba-9ddb-7c1c8b526f5b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c16d23b1175f46e1b69a844ee82c5295'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:21.837855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6191f963-e3e8-44ba-9ddb-7c1c8b526f5b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c16d23b1175f46e1b69a844ee82c5295'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:21.846385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 856/1710] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:21 2022] DELETE /identity/v3/users/c16d23b1175f46e1b69a844ee82c5295 => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:21.848292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6191f963-e3e8-44ba-9ddb-7c1c8b526f5b admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:21.852143 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-6191f963-e3e8-44ba-9ddb-7c1c8b526f5b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:21.984034 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f098df-f435-48c9-a831-0e450ef6ef03 service nova] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:22.053188 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6191f963-e3e8-44ba-9ddb-7c1c8b526f5b admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:22.075700 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490e6256-dfae-4969-bd6c-7e6e98af97ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:22.076473 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490e6256-dfae-4969-bd6c-7e6e98af97ef admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:22.076829 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490e6256-dfae-4969-bd6c-7e6e98af97ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:22.077230 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490e6256-dfae-4969-bd6c-7e6e98af97ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:22.077946 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490e6256-dfae-4969-bd6c-7e6e98af97ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:22.079687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490e6256-dfae-4969-bd6c-7e6e98af97ef admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:22.093133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 857/1711] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:21 2022] GET /identity/v3/roles => generated 1420 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:22.095947 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c832cefa-afd0-4f15-bb87-9e7328f78782 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:22.095947 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c832cefa-afd0-4f15-bb87-9e7328f78782 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:22.095947 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c832cefa-afd0-4f15-bb87-9e7328f78782 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:22.132468 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c832cefa-afd0-4f15-bb87-9e7328f78782 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:22.134450 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c832cefa-afd0-4f15-bb87-9e7328f78782 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:22.134728 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8eee01f-f510-4e73-bb83-89729e857e8c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:22.135770 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8eee01f-f510-4e73-bb83-89729e857e8c service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:22.138731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8eee01f-f510-4e73-bb83-89729e857e8c service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:22.139330 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8eee01f-f510-4e73-bb83-89729e857e8c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:22.144447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e8eee01f-f510-4e73-bb83-89729e857e8c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:22.153813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 858/1712] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:22.154757 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c832cefa-afd0-4f15-bb87-9e7328f78782 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:22.174481 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c832cefa-afd0-4f15-bb87-9e7328f78782 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:22.187973 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8eee01f-f510-4e73-bb83-89729e857e8c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:22.191402 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8eee01f-f510-4e73-bb83-89729e857e8c service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:22.192849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba58795-80b2-46a5-a8e9-122b61489688 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:22.193593 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba58795-80b2-46a5-a8e9-122b61489688 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:22.194582 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba58795-80b2-46a5-a8e9-122b61489688 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:22.194855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba58795-80b2-46a5-a8e9-122b61489688 admin admin] PATH_INFO: `/v3/projects/da4abb67ad4d4464b74e5ccbbd09b426/users/3432bae1c2ed4f6fa40333efc65f971c/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:22.197448 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba58795-80b2-46a5-a8e9-122b61489688 admin admin] RBAC: Authorizing `identity:create_grant(project_id=da4abb67ad4d4464b74e5ccbbd09b426, user_id=3432bae1c2ed4f6fa40333efc65f971c, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:22.205712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba58795-80b2-46a5-a8e9-122b61489688 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:22.228868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 855/1713] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:21 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:22.231863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8eee01f-f510-4e73-bb83-89729e857e8c service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:22.242006 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 859/1714] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:22 2022] PUT /identity/v3/projects/da4abb67ad4d4464b74e5ccbbd09b426/users/3432bae1c2ed4f6fa40333efc65f971c/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:22.242193 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e8eee01f-f510-4e73-bb83-89729e857e8c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:22.260273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8eee01f-f510-4e73-bb83-89729e857e8c service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:22.332502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7708c614-e980-4588-b59a-b1feadf7fbf0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:22.333163 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7708c614-e980-4588-b59a-b1feadf7fbf0 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:22.333554 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7708c614-e980-4588-b59a-b1feadf7fbf0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:22.334070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7708c614-e980-4588-b59a-b1feadf7fbf0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:22.467872 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7708c614-e980-4588-b59a-b1feadf7fbf0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:22.472267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7708c614-e980-4588-b59a-b1feadf7fbf0 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:22.487148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 856/1715] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 257 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:23.015167 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:24.248110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba58795-80b2-46a5-a8e9-122b61489688 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:24.249934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-1ba58795-80b2-46a5-a8e9-122b61489688 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:24.289868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba58795-80b2-46a5-a8e9-122b61489688 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:24.303459 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd5635e-732c-45df-b933-dc97a34f593a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:24.304105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd5635e-732c-45df-b933-dc97a34f593a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:24.305606 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd5635e-732c-45df-b933-dc97a34f593a admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:24.305880 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd5635e-732c-45df-b933-dc97a34f593a admin admin] PATH_INFO: `/v3/projects/a011b7f9815648bbbd033072449a4b5f` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:24.321357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd5635e-732c-45df-b933-dc97a34f593a admin admin] RBAC: Authorizing `identity:delete_project(project_id=a011b7f9815648bbbd033072449a4b5f)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:24.324060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd5635e-732c-45df-b933-dc97a34f593a admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:24.520469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-dfd5635e-732c-45df-b933-dc97a34f593a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a011b7f9815648bbbd033072449a4b5f', 'request_id': 'req-dfd5635e-732c-45df-b933-dc97a34f593a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:24.627175 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-dfd5635e-732c-45df-b933-dc97a34f593a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a011b7f9815648bbbd033072449a4b5f', 'request_id': 'req-dfd5635e-732c-45df-b933-dc97a34f593a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:24.640554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-dfd5635e-732c-45df-b933-dc97a34f593a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a011b7f9815648bbbd033072449a4b5f', 'request_id': 'req-dfd5635e-732c-45df-b933-dc97a34f593a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:24.684492 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-dfd5635e-732c-45df-b933-dc97a34f593a admin admin] The token cache is being invalidate because project a011b7f9815648bbbd033072449a4b5f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:24.684786 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-dfd5635e-732c-45df-b933-dc97a34f593a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:24.688556 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-dfd5635e-732c-45df-b933-dc97a34f593a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:24.693523 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 860/1716] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:24 2022] DELETE /identity/v3/projects/a011b7f9815648bbbd033072449a4b5f => generated 0 bytes in 443 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:24.695424 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7708c614-e980-4588-b59a-b1feadf7fbf0 service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:24.714183 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7708c614-e980-4588-b59a-b1feadf7fbf0 service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:24.810004 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67995548-f4ed-4524-bde9-058a59da3a0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:24.810664 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67995548-f4ed-4524-bde9-058a59da3a0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:24.810989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67995548-f4ed-4524-bde9-058a59da3a0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:24.811320 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67995548-f4ed-4524-bde9-058a59da3a0c admin admin] PATH_INFO: `/v3/users/b8d762b3372341ba86f26bdf30cb3bb3` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:24.812721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67995548-f4ed-4524-bde9-058a59da3a0c admin admin] RBAC: Authorizing `identity:delete_user(user_id=b8d762b3372341ba86f26bdf30cb3bb3)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:24.815350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67995548-f4ed-4524-bde9-058a59da3a0c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:25.131956 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-67995548-f4ed-4524-bde9-058a59da3a0c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b8d762b3372341ba86f26bdf30cb3bb3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:25.235519 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-67995548-f4ed-4524-bde9-058a59da3a0c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b8d762b3372341ba86f26bdf30cb3bb3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:25.236359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-67995548-f4ed-4524-bde9-058a59da3a0c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b8d762b3372341ba86f26bdf30cb3bb3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:25.286407 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-67995548-f4ed-4524-bde9-058a59da3a0c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b8d762b3372341ba86f26bdf30cb3bb3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:25.303607 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2cb9f4-4e86-49f9-9406-853174d604c4 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:25.304156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2cb9f4-4e86-49f9-9406-853174d604c4 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:25.304156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2cb9f4-4e86-49f9-9406-853174d604c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:25.334739 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:25.342763 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-67995548-f4ed-4524-bde9-058a59da3a0c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b8d762b3372341ba86f26bdf30cb3bb3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:25.402211 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 857/1717] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:24 2022] DELETE /identity/v3/users/b8d762b3372341ba86f26bdf30cb3bb3 => generated 0 bytes in 707 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:25.440314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-9f2cb9f4-4e86-49f9-9406-853174d604c4 None None] MFA Rules not processed for user `3432bae1c2ed4f6fa40333efc65f971c`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:25.441684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-9f2cb9f4-4e86-49f9-9406-853174d604c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:25.528429 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:25.584504 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 861/1718] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:25 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:26.123839 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb2c461-f7db-401c-b5a3-83eb6e3d9d4c None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:26.124353 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb2c461-f7db-401c-b5a3-83eb6e3d9d4c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:26.124428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb2c461-f7db-401c-b5a3-83eb6e3d9d4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:26.172623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-bcb2c461-f7db-401c-b5a3-83eb6e3d9d4c None None] MFA Rules not processed for user `0db30f990aae49fabf9b8b19cd019de9`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:26.176285 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-bcb2c461-f7db-401c-b5a3-83eb6e3d9d4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:26.231582 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2cb9f4-4e86-49f9-9406-853174d604c4 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:26.240914 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-9f2cb9f4-4e86-49f9-9406-853174d604c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:26.341918 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 858/1719] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:26 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:26.541953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2cb9f4-4e86-49f9-9406-853174d604c4 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:26.557399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd6feac-e97a-4d79-8a4c-570aa59dd82b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:26.558056 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd6feac-e97a-4d79-8a4c-570aa59dd82b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:26.558379 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd6feac-e97a-4d79-8a4c-570aa59dd82b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:26.559464 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd6feac-e97a-4d79-8a4c-570aa59dd82b admin admin] PATH_INFO: `/v3/projects/e49858138ae041228ad1176c68843484` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:26.561259 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd6feac-e97a-4d79-8a4c-570aa59dd82b admin admin] RBAC: Authorizing `identity:delete_project(project_id=e49858138ae041228ad1176c68843484)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:26.563552 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd6feac-e97a-4d79-8a4c-570aa59dd82b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:26.863840 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9fd6feac-e97a-4d79-8a4c-570aa59dd82b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e49858138ae041228ad1176c68843484', 'request_id': 'req-9fd6feac-e97a-4d79-8a4c-570aa59dd82b'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:26.947444 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9fd6feac-e97a-4d79-8a4c-570aa59dd82b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e49858138ae041228ad1176c68843484', 'request_id': 'req-9fd6feac-e97a-4d79-8a4c-570aa59dd82b'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:26.959065 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9fd6feac-e97a-4d79-8a4c-570aa59dd82b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e49858138ae041228ad1176c68843484', 'request_id': 'req-9fd6feac-e97a-4d79-8a4c-570aa59dd82b'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:27.016960 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9fd6feac-e97a-4d79-8a4c-570aa59dd82b admin admin] The token cache is being invalidate because project e49858138ae041228ad1176c68843484 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:27.017117 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9fd6feac-e97a-4d79-8a4c-570aa59dd82b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:27.022171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9fd6feac-e97a-4d79-8a4c-570aa59dd82b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:27.025208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 862/1720] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:26 2022] DELETE /identity/v3/projects/e49858138ae041228ad1176c68843484 => generated 0 bytes in 794 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:27.030605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2049972c-2f30-4562-90c7-60949368c8f8 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:27.031048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2049972c-2f30-4562-90c7-60949368c8f8 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:27.031519 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2049972c-2f30-4562-90c7-60949368c8f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:27.087661 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-2049972c-2f30-4562-90c7-60949368c8f8 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:27.089026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2049972c-2f30-4562-90c7-60949368c8f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:27.260031 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 859/1721] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:27.275637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf85201-32df-42de-8e8a-e67ee6f4a7f4 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:27.275637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf85201-32df-42de-8e8a-e67ee6f4a7f4 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:27.275637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf85201-32df-42de-8e8a-e67ee6f4a7f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:27.292153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc14a0e0-a2b9-4711-bb9d-8f66b7c367e9 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:27.293280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc14a0e0-a2b9-4711-bb9d-8f66b7c367e9 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:27.293575 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc14a0e0-a2b9-4711-bb9d-8f66b7c367e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:27.324586 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-1bf85201-32df-42de-8e8a-e67ee6f4a7f4 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:27.330757 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-1bf85201-32df-42de-8e8a-e67ee6f4a7f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:27.361412 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 863/1722] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:27.369801 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf85201-32df-42de-8e8a-e67ee6f4a7f4 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:27.388762 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-dc14a0e0-a2b9-4711-bb9d-8f66b7c367e9 None None] MFA Rules not processed for user `b471d768cd814ff8b4f6bd35d21e24a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:27.390319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-dc14a0e0-a2b9-4711-bb9d-8f66b7c367e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:27.402002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf85201-32df-42de-8e8a-e67ee6f4a7f4 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:27.415365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d2b65b-6838-45ea-9b86-d259a059fdf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:27.430663 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d2b65b-6838-45ea-9b86-d259a059fdf8 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:27.430962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d2b65b-6838-45ea-9b86-d259a059fdf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:27.431242 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d2b65b-6838-45ea-9b86-d259a059fdf8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:27.431934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d2b65b-6838-45ea-9b86-d259a059fdf8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:27.437864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d2b65b-6838-45ea-9b86-d259a059fdf8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:27.448100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d2b65b-6838-45ea-9b86-d259a059fdf8 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:27.523084 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 864/1723] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:02:27.527365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d2b65b-6838-45ea-9b86-d259a059fdf8 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:27.548035 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 860/1724] 10.222.0.62 () {58 vars in 1009 bytes} [Fri Sep 23 14:02:27 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:02:27.558877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d2b65b-6838-45ea-9b86-d259a059fdf8 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:27.569481 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9101a75-a1f0-411f-b738-45fa8a3a4b86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:27.570870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9101a75-a1f0-411f-b738-45fa8a3a4b86 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:27.571298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9101a75-a1f0-411f-b738-45fa8a3a4b86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:27.571773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9101a75-a1f0-411f-b738-45fa8a3a4b86 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:27.572459 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9101a75-a1f0-411f-b738-45fa8a3a4b86 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:27.581370 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9101a75-a1f0-411f-b738-45fa8a3a4b86 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:27.711394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 865/1725] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:27 2022] POST /identity/v3/projects => generated 365 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:27.718527 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc14a0e0-a2b9-4711-bb9d-8f66b7c367e9 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:27.740398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc14a0e0-a2b9-4711-bb9d-8f66b7c367e9 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:27.749379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:27.772045 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9101a75-a1f0-411f-b738-45fa8a3a4b86 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:27.778108 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a9101a75-a1f0-411f-b738-45fa8a3a4b86 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:27.837476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e77957c-8955-4f3f-81a0-985862147c55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:27.838206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e77957c-8955-4f3f-81a0-985862147c55 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:27.838684 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e77957c-8955-4f3f-81a0-985862147c55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:27.839028 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e77957c-8955-4f3f-81a0-985862147c55 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:27.839764 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e77957c-8955-4f3f-81a0-985862147c55 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:27.847082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e77957c-8955-4f3f-81a0-985862147c55 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:27.962677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9101a75-a1f0-411f-b738-45fa8a3a4b86 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:27.970320 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f98aed-d7e7-4b46-b774-0c16f65b120b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:27.971329 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f98aed-d7e7-4b46-b774-0c16f65b120b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:27.971721 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f98aed-d7e7-4b46-b774-0c16f65b120b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:27.972087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f98aed-d7e7-4b46-b774-0c16f65b120b admin admin] PATH_INFO: `/v3/users/6d5d064339d14cd09569765c3ce9b0f3` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:27.973913 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f98aed-d7e7-4b46-b774-0c16f65b120b admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d5d064339d14cd09569765c3ce9b0f3)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:27.976667 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f98aed-d7e7-4b46-b774-0c16f65b120b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:28.063838 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 861/1726] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:27 2022] POST /identity/v3/users => generated 283 bytes in 346 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:28.065273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e77957c-8955-4f3f-81a0-985862147c55 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:28.067569 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3e77957c-8955-4f3f-81a0-985862147c55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:28.208471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e77957c-8955-4f3f-81a0-985862147c55 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:28.223464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd2bfee-6f0d-4567-a87e-8dbc279da4e3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b9269509f1d454eab7a3008c2ba0039', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:28.225872 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd2bfee-6f0d-4567-a87e-8dbc279da4e3 service swift] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:28.226310 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd2bfee-6f0d-4567-a87e-8dbc279da4e3 service swift] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:28.226817 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd2bfee-6f0d-4567-a87e-8dbc279da4e3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:28.271926 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-02f98aed-d7e7-4b46-b774-0c16f65b120b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d5d064339d14cd09569765c3ce9b0f3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:28.359160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd2bfee-6f0d-4567-a87e-8dbc279da4e3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:28.360740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd2bfee-6f0d-4567-a87e-8dbc279da4e3 service swift] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:28.375281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-02f98aed-d7e7-4b46-b774-0c16f65b120b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d5d064339d14cd09569765c3ce9b0f3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:28.379772 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 862/1727] 10.222.0.62 () {60 vars in 1384 bytes} [Fri Sep 23 14:02:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 23 14:02:28.383899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd2bfee-6f0d-4567-a87e-8dbc279da4e3 service swift] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:28.401535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-02f98aed-d7e7-4b46-b774-0c16f65b120b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d5d064339d14cd09569765c3ce9b0f3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:28.402281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-02f98aed-d7e7-4b46-b774-0c16f65b120b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d5d064339d14cd09569765c3ce9b0f3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:28.405051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd2bfee-6f0d-4567-a87e-8dbc279da4e3 service swift] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:28.411231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d90a445-10a1-4af5-ba62-055d110cb4e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:28.412655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d90a445-10a1-4af5-ba62-055d110cb4e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:28.413059 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d90a445-10a1-4af5-ba62-055d110cb4e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:28.413553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d90a445-10a1-4af5-ba62-055d110cb4e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:28.414561 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d90a445-10a1-4af5-ba62-055d110cb4e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:28.419619 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-02f98aed-d7e7-4b46-b774-0c16f65b120b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d5d064339d14cd09569765c3ce9b0f3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:28.420326 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d90a445-10a1-4af5-ba62-055d110cb4e2 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:28.434803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 866/1728] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:27 2022] DELETE /identity/v3/users/6d5d064339d14cd09569765c3ce9b0f3 => generated 0 bytes in 661 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:28.435480 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30eee38c-f29a-49bd-a65c-23de33015f75 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:28.438468 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30eee38c-f29a-49bd-a65c-23de33015f75 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:28.438685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 863/1729] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:28 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:28.438771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30eee38c-f29a-49bd-a65c-23de33015f75 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:28.438869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d90a445-10a1-4af5-ba62-055d110cb4e2 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:28.439564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 867/1730] 10.222.0.62 () {54 vars in 921 bytes} [Fri Sep 23 14:02:28 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:02:28.443650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8d90a445-10a1-4af5-ba62-055d110cb4e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:28.447321 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a127cba-0cb0-449f-8607-34e686883031 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:28.447966 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a127cba-0cb0-449f-8607-34e686883031 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:28.448244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a127cba-0cb0-449f-8607-34e686883031 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:28.472974 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d90a445-10a1-4af5-ba62-055d110cb4e2 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:28.496597 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-6a127cba-0cb0-449f-8607-34e686883031 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:28.507109 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-6a127cba-0cb0-449f-8607-34e686883031 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:28.584788 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e6788a-6333-42e9-9cd4-b63b2a060d3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:28.585486 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e6788a-6333-42e9-9cd4-b63b2a060d3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:28.585751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e6788a-6333-42e9-9cd4-b63b2a060d3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:28.586042 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e6788a-6333-42e9-9cd4-b63b2a060d3a admin admin] PATH_INFO: `/v3/users/2687ba3a77d64bb99bbe6bd38b9b63a2` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:28.587295 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e6788a-6333-42e9-9cd4-b63b2a060d3a admin admin] RBAC: Authorizing `identity:delete_user(user_id=2687ba3a77d64bb99bbe6bd38b9b63a2)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:28.591154 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e6788a-6333-42e9-9cd4-b63b2a060d3a admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:28.612322 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:28.667731 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 868/1731] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:28.668831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a127cba-0cb0-449f-8607-34e686883031 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:28.672924 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-6a127cba-0cb0-449f-8607-34e686883031 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:28.699779 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a127cba-0cb0-449f-8607-34e686883031 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:28.705889 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109edac8-894c-4a58-bd35-36fd087a739d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:28.706698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109edac8-894c-4a58-bd35-36fd087a739d admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:28.706977 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109edac8-894c-4a58-bd35-36fd087a739d admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:28.707251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109edac8-894c-4a58-bd35-36fd087a739d admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:28.707929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109edac8-894c-4a58-bd35-36fd087a739d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:28.710857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109edac8-894c-4a58-bd35-36fd087a739d admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:28.726825 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 869/1732] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:28 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:28.728228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248a0641-9953-4cae-8b8a-e2177543ca87 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:28.728518 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248a0641-9953-4cae-8b8a-e2177543ca87 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:28.728787 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248a0641-9953-4cae-8b8a-e2177543ca87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:28.761450 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-248a0641-9953-4cae-8b8a-e2177543ca87 None None] MFA Rules not processed for user `ce249d4a99c34891b6ba941f57a14dd4`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:28.762796 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-248a0641-9953-4cae-8b8a-e2177543ca87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:28.835505 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f4e6788a-6333-42e9-9cd4-b63b2a060d3a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2687ba3a77d64bb99bbe6bd38b9b63a2'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:28.855709 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 870/1733] 10.222.0.62 () {58 vars in 1018 bytes} [Fri Sep 23 14:02:28 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:28.856679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248a0641-9953-4cae-8b8a-e2177543ca87 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:28.858110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-248a0641-9953-4cae-8b8a-e2177543ca87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:28.907639 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:28.931514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248a0641-9953-4cae-8b8a-e2177543ca87 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:28.934265 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f4e6788a-6333-42e9-9cd4-b63b2a060d3a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2687ba3a77d64bb99bbe6bd38b9b63a2'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:28.935015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f4e6788a-6333-42e9-9cd4-b63b2a060d3a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2687ba3a77d64bb99bbe6bd38b9b63a2'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:28.935264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01745d1-900a-4201-8a0d-e201a2ecc8b7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:28.935920 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01745d1-900a-4201-8a0d-e201a2ecc8b7 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:28.936368 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01745d1-900a-4201-8a0d-e201a2ecc8b7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:28.936463 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01745d1-900a-4201-8a0d-e201a2ecc8b7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:28.938424 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c01745d1-900a-4201-8a0d-e201a2ecc8b7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:28.952570 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01745d1-900a-4201-8a0d-e201a2ecc8b7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:28.952746 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f4e6788a-6333-42e9-9cd4-b63b2a060d3a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2687ba3a77d64bb99bbe6bd38b9b63a2'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:28.954217 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01745d1-900a-4201-8a0d-e201a2ecc8b7 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:28.960253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f4e6788a-6333-42e9-9cd4-b63b2a060d3a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2687ba3a77d64bb99bbe6bd38b9b63a2'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:28.965556 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 871/1734] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:28.966166 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01745d1-900a-4201-8a0d-e201a2ecc8b7 service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:28.967638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c01745d1-900a-4201-8a0d-e201a2ecc8b7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:28.973471 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 864/1735] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:28 2022] DELETE /identity/v3/users/2687ba3a77d64bb99bbe6bd38b9b63a2 => generated 0 bytes in 533 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:28.975962 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e6788a-6333-42e9-9cd4-b63b2a060d3a admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:28.977492 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f4e6788a-6333-42e9-9cd4-b63b2a060d3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:28.984512 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01745d1-900a-4201-8a0d-e201a2ecc8b7 service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:29.049442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944c405b-2c86-49d6-86cf-fe37d28eaa9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:29.049718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944c405b-2c86-49d6-86cf-fe37d28eaa9a admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:29.050099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944c405b-2c86-49d6-86cf-fe37d28eaa9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:29.050409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944c405b-2c86-49d6-86cf-fe37d28eaa9a admin admin] PATH_INFO: `/v3/projects/59a3cb3df9fe443f90dfd489e581278c/users/83e5475d545c418cb1b1cb0a18447510/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:29.063825 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e6788a-6333-42e9-9cd4-b63b2a060d3a admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:29.069852 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-944c405b-2c86-49d6-86cf-fe37d28eaa9a admin admin] RBAC: Authorizing `identity:create_grant(project_id=59a3cb3df9fe443f90dfd489e581278c, user_id=83e5475d545c418cb1b1cb0a18447510, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:29.072320 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d27cc0-d1ab-4705-8874-42706b04f909 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce249d4a99c34891b6ba941f57a14dd4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:29.073181 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d27cc0-d1ab-4705-8874-42706b04f909 service placement] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:29.073638 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d27cc0-d1ab-4705-8874-42706b04f909 service placement] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:29.073741 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-944c405b-2c86-49d6-86cf-fe37d28eaa9a admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:29.074828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d27cc0-d1ab-4705-8874-42706b04f909 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:29.077339 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-60d27cc0-d1ab-4705-8874-42706b04f909 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:29.110869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d27cc0-d1ab-4705-8874-42706b04f909 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:29.114753 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d27cc0-d1ab-4705-8874-42706b04f909 service placement] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:29.135442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 865/1736] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:28 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:29.136589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88167eab-cedb-481a-adbc-e6c6321b4960 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:29.136878 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88167eab-cedb-481a-adbc-e6c6321b4960 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:29.139131 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88167eab-cedb-481a-adbc-e6c6321b4960 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:29.163415 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 872/1737] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:28 2022] PUT /identity/v3/projects/59a3cb3df9fe443f90dfd489e581278c/users/83e5475d545c418cb1b1cb0a18447510/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:29.172748 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-88167eab-cedb-481a-adbc-e6c6321b4960 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:29.174101 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-88167eab-cedb-481a-adbc-e6c6321b4960 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:29.175352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c300d55e-e753-4b53-84ef-fb61dc7a36d7 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:29.175792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c300d55e-e753-4b53-84ef-fb61dc7a36d7 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:29.176411 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c300d55e-e753-4b53-84ef-fb61dc7a36d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:29.268478 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c300d55e-e753-4b53-84ef-fb61dc7a36d7 None None] MFA Rules not processed for user `83e5475d545c418cb1b1cb0a18447510`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:29.271912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c300d55e-e753-4b53-84ef-fb61dc7a36d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:29.307379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 866/1738] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:29.350982 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88167eab-cedb-481a-adbc-e6c6321b4960 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:29.352424 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-88167eab-cedb-481a-adbc-e6c6321b4960 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:29.472648 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 873/1739] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:29 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 300 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:29.486877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c300d55e-e753-4b53-84ef-fb61dc7a36d7 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:29.511714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88167eab-cedb-481a-adbc-e6c6321b4960 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:29.517824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c300d55e-e753-4b53-84ef-fb61dc7a36d7 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:29.527432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb1fabd-a8af-4d14-bd8a-810b8ff63a60 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:29.528079 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb1fabd-a8af-4d14-bd8a-810b8ff63a60 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:29.528365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb1fabd-a8af-4d14-bd8a-810b8ff63a60 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:29.528640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb1fabd-a8af-4d14-bd8a-810b8ff63a60 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:29.532281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc4375b-fef1-4d93-ad36-9a9d4d08b47a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:29.532996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc4375b-fef1-4d93-ad36-9a9d4d08b47a admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:29.533276 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc4375b-fef1-4d93-ad36-9a9d4d08b47a admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:29.533547 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc4375b-fef1-4d93-ad36-9a9d4d08b47a admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:29.534193 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc4375b-fef1-4d93-ad36-9a9d4d08b47a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:29.540839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc4375b-fef1-4d93-ad36-9a9d4d08b47a admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:29.547144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb1fabd-a8af-4d14-bd8a-810b8ff63a60 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:29.549134 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb1fabd-a8af-4d14-bd8a-810b8ff63a60 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:29.567205 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 867/1740] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:02:29.609177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb1fabd-a8af-4d14-bd8a-810b8ff63a60 service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:29.610926 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-1eb1fabd-a8af-4d14-bd8a-810b8ff63a60 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:29.626252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb1fabd-a8af-4d14-bd8a-810b8ff63a60 service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:29.635161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6bd8c7-10dd-4d5b-b0a2-d4b58ea2a29c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:29.635161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6bd8c7-10dd-4d5b-b0a2-d4b58ea2a29c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:29.635161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6bd8c7-10dd-4d5b-b0a2-d4b58ea2a29c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:29.635161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6bd8c7-10dd-4d5b-b0a2-d4b58ea2a29c admin admin] PATH_INFO: `/v3/projects/fd25990a10694a5aa7debfcae216d00c` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:29.635587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6bd8c7-10dd-4d5b-b0a2-d4b58ea2a29c admin admin] RBAC: Authorizing `identity:delete_project(project_id=fd25990a10694a5aa7debfcae216d00c)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:29.636431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c6bd8c7-10dd-4d5b-b0a2-d4b58ea2a29c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:29.666171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 874/1741] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:29 2022] POST /identity/v3/projects => generated 361 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:29.669960 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc4375b-fef1-4d93-ad36-9a9d4d08b47a admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:29.733879 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc4375b-fef1-4d93-ad36-9a9d4d08b47a admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:29.738545 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4312b43d-3d74-49ae-aa51-719fab5222bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:29.739148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4312b43d-3d74-49ae-aa51-719fab5222bd admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:29.739424 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4312b43d-3d74-49ae-aa51-719fab5222bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:29.739693 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4312b43d-3d74-49ae-aa51-719fab5222bd admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:29.740540 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4312b43d-3d74-49ae-aa51-719fab5222bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:29.742483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4312b43d-3d74-49ae-aa51-719fab5222bd admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:29.797664 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-4c6bd8c7-10dd-4d5b-b0a2-d4b58ea2a29c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fd25990a10694a5aa7debfcae216d00c', 'request_id': 'req-4c6bd8c7-10dd-4d5b-b0a2-d4b58ea2a29c'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:29.869875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 875/1742] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:29 2022] POST /identity/v3/users => generated 287 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:02:29.870637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4312b43d-3d74-49ae-aa51-719fab5222bd admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:29.872307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-4312b43d-3d74-49ae-aa51-719fab5222bd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:29.885659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4312b43d-3d74-49ae-aa51-719fab5222bd admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:29.899877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-4c6bd8c7-10dd-4d5b-b0a2-d4b58ea2a29c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fd25990a10694a5aa7debfcae216d00c', 'request_id': 'req-4c6bd8c7-10dd-4d5b-b0a2-d4b58ea2a29c'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:29.907328 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-4c6bd8c7-10dd-4d5b-b0a2-d4b58ea2a29c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fd25990a10694a5aa7debfcae216d00c', 'request_id': 'req-4c6bd8c7-10dd-4d5b-b0a2-d4b58ea2a29c'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:29.931522 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-4c6bd8c7-10dd-4d5b-b0a2-d4b58ea2a29c admin admin] The token cache is being invalidate because project fd25990a10694a5aa7debfcae216d00c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:29.932086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-4c6bd8c7-10dd-4d5b-b0a2-d4b58ea2a29c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:29.933619 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-4c6bd8c7-10dd-4d5b-b0a2-d4b58ea2a29c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:29.937265 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 868/1743] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:29 2022] DELETE /identity/v3/projects/fd25990a10694a5aa7debfcae216d00c => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:02:29.938591 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6bd8c7-10dd-4d5b-b0a2-d4b58ea2a29c admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:29.940912 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-4c6bd8c7-10dd-4d5b-b0a2-d4b58ea2a29c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:30.006732 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54d7957-9fec-4c19-a947-652f7fe23272 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:30.007320 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54d7957-9fec-4c19-a947-652f7fe23272 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:30.007598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54d7957-9fec-4c19-a947-652f7fe23272 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:30.007872 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54d7957-9fec-4c19-a947-652f7fe23272 admin admin] PATH_INFO: `/v3/users/be681f850a2c4f1f93415ea592eba15a` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:30.011170 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54d7957-9fec-4c19-a947-652f7fe23272 admin admin] RBAC: Authorizing `identity:delete_user(user_id=be681f850a2c4f1f93415ea592eba15a)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:30.012886 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54d7957-9fec-4c19-a947-652f7fe23272 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:30.051227 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6bd8c7-10dd-4d5b-b0a2-d4b58ea2a29c admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:30.057110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d94d444-21c0-4296-8a70-88507df9b75c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:30.057709 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d94d444-21c0-4296-8a70-88507df9b75c admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:30.058021 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d94d444-21c0-4296-8a70-88507df9b75c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:30.058289 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d94d444-21c0-4296-8a70-88507df9b75c admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:30.058975 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d94d444-21c0-4296-8a70-88507df9b75c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:30.060557 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d94d444-21c0-4296-8a70-88507df9b75c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:30.071123 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 869/1744] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:29 2022] GET /identity/v3/roles => generated 1420 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:30.071934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d94d444-21c0-4296-8a70-88507df9b75c admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:30.073552 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-7d94d444-21c0-4296-8a70-88507df9b75c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:30.085904 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d94d444-21c0-4296-8a70-88507df9b75c admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:30.091355 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a355273f-ad0e-4aa8-bf8b-0c429d73ee21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:30.092009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a355273f-ad0e-4aa8-bf8b-0c429d73ee21 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:30.092317 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a355273f-ad0e-4aa8-bf8b-0c429d73ee21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:30.092585 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a355273f-ad0e-4aa8-bf8b-0c429d73ee21 admin admin] PATH_INFO: `/v3/users/80aaa35994a54ce5ac2612217c6d7734` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:30.093648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a355273f-ad0e-4aa8-bf8b-0c429d73ee21 admin admin] RBAC: Authorizing `identity:delete_user(user_id=80aaa35994a54ce5ac2612217c6d7734)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:30.096338 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a355273f-ad0e-4aa8-bf8b-0c429d73ee21 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:30.287386 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e54d7957-9fec-4c19-a947-652f7fe23272 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'be681f850a2c4f1f93415ea592eba15a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:30.308525 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:30.327363 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a355273f-ad0e-4aa8-bf8b-0c429d73ee21 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80aaa35994a54ce5ac2612217c6d7734'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:30.346145 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e54d7957-9fec-4c19-a947-652f7fe23272 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'be681f850a2c4f1f93415ea592eba15a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:30.363679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e54d7957-9fec-4c19-a947-652f7fe23272 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'be681f850a2c4f1f93415ea592eba15a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:30.365643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e54d7957-9fec-4c19-a947-652f7fe23272 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'be681f850a2c4f1f93415ea592eba15a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:30.378257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e54d7957-9fec-4c19-a947-652f7fe23272 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'be681f850a2c4f1f93415ea592eba15a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:30.387052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 876/1745] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:29 2022] DELETE /identity/v3/users/be681f850a2c4f1f93415ea592eba15a => generated 0 bytes in 517 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:30.391013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54d7957-9fec-4c19-a947-652f7fe23272 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:30.392431 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-e54d7957-9fec-4c19-a947-652f7fe23272 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:30.397272 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a355273f-ad0e-4aa8-bf8b-0c429d73ee21 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80aaa35994a54ce5ac2612217c6d7734'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:30.398024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a355273f-ad0e-4aa8-bf8b-0c429d73ee21 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80aaa35994a54ce5ac2612217c6d7734'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:30.423740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a355273f-ad0e-4aa8-bf8b-0c429d73ee21 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80aaa35994a54ce5ac2612217c6d7734'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:30.475256 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a355273f-ad0e-4aa8-bf8b-0c429d73ee21 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80aaa35994a54ce5ac2612217c6d7734'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:30.518303 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 870/1746] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:30 2022] DELETE /identity/v3/users/80aaa35994a54ce5ac2612217c6d7734 => generated 0 bytes in 445 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:30.519313 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a355273f-ad0e-4aa8-bf8b-0c429d73ee21 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:30.523051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a355273f-ad0e-4aa8-bf8b-0c429d73ee21 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:30.554928 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54d7957-9fec-4c19-a947-652f7fe23272 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:30.578757 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:30.714734 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15daa1f3-d4ce-4f29-ad74-4db21a4b75b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:30.716198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15daa1f3-d4ce-4f29-ad74-4db21a4b75b4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:30.716618 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15daa1f3-d4ce-4f29-ad74-4db21a4b75b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:30.716838 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a355273f-ad0e-4aa8-bf8b-0c429d73ee21 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:30.718102 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15daa1f3-d4ce-4f29-ad74-4db21a4b75b4 admin admin] PATH_INFO: `/v3/projects/59c7d00de35544e1ad06e8b4f290e6c0/users/ac803da0c82141a78b9093c1e360dafe/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:30.746300 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec97337-2e95-43a6-93e6-345b4e4f52a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:30.749756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec97337-2e95-43a6-93e6-345b4e4f52a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:30.749756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec97337-2e95-43a6-93e6-345b4e4f52a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:30.751438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec97337-2e95-43a6-93e6-345b4e4f52a2 admin admin] PATH_INFO: `/v3/projects/4a25811b83194cbf820eb25171f370a6` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:30.755172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec97337-2e95-43a6-93e6-345b4e4f52a2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4a25811b83194cbf820eb25171f370a6)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:30.757781 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15daa1f3-d4ce-4f29-ad74-4db21a4b75b4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=59c7d00de35544e1ad06e8b4f290e6c0, user_id=ac803da0c82141a78b9093c1e360dafe, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:30.759003 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec97337-2e95-43a6-93e6-345b4e4f52a2 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:30.769594 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15daa1f3-d4ce-4f29-ad74-4db21a4b75b4 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:30.889275 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 877/1747] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:30 2022] PUT /identity/v3/projects/59c7d00de35544e1ad06e8b4f290e6c0/users/ac803da0c82141a78b9093c1e360dafe/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 499 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:30.890248 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15daa1f3-d4ce-4f29-ad74-4db21a4b75b4 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:30.893524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-15daa1f3-d4ce-4f29-ad74-4db21a4b75b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:30.942054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-eec97337-2e95-43a6-93e6-345b4e4f52a2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4a25811b83194cbf820eb25171f370a6', 'request_id': 'req-eec97337-2e95-43a6-93e6-345b4e4f52a2'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:31.031525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15daa1f3-d4ce-4f29-ad74-4db21a4b75b4 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:31.084596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-eec97337-2e95-43a6-93e6-345b4e4f52a2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4a25811b83194cbf820eb25171f370a6', 'request_id': 'req-eec97337-2e95-43a6-93e6-345b4e4f52a2'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:31.103581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-eec97337-2e95-43a6-93e6-345b4e4f52a2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4a25811b83194cbf820eb25171f370a6', 'request_id': 'req-eec97337-2e95-43a6-93e6-345b4e4f52a2'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:31.135419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbf667a-46f1-45d6-b769-3d23e7946b4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:31.136052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbf667a-46f1-45d6-b769-3d23e7946b4c admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:31.136355 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbf667a-46f1-45d6-b769-3d23e7946b4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:31.136596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbf667a-46f1-45d6-b769-3d23e7946b4c admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:31.137246 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbf667a-46f1-45d6-b769-3d23e7946b4c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:31.139196 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbf667a-46f1-45d6-b769-3d23e7946b4c admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:31.156576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 878/1748] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:30 2022] GET /identity/v3/roles => generated 1420 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:31.156576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc672c0-85b7-4792-ac79-b7b908123a48 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:31.156576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc672c0-85b7-4792-ac79-b7b908123a48 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:31.156576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc672c0-85b7-4792-ac79-b7b908123a48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:31.177714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-eec97337-2e95-43a6-93e6-345b4e4f52a2 admin admin] The token cache is being invalidate because project 4a25811b83194cbf820eb25171f370a6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:31.177714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-eec97337-2e95-43a6-93e6-345b4e4f52a2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:31.180291 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-eec97337-2e95-43a6-93e6-345b4e4f52a2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:31.186121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 871/1749] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:30 2022] DELETE /identity/v3/projects/4a25811b83194cbf820eb25171f370a6 => generated 0 bytes in 669 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:31.187495 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec97337-2e95-43a6-93e6-345b4e4f52a2 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:31.189015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-eec97337-2e95-43a6-93e6-345b4e4f52a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:31.190744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-3dc672c0-85b7-4792-ac79-b7b908123a48 None None] MFA Rules not processed for user `580a35627c234a25aa04f5d376177ca5`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:31.192058 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-3dc672c0-85b7-4792-ac79-b7b908123a48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:31.243070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec97337-2e95-43a6-93e6-345b4e4f52a2 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:31.250391 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11968277-91f3-4952-81b0-ef4fedc44fcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:31.252348 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11968277-91f3-4952-81b0-ef4fedc44fcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:31.252876 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11968277-91f3-4952-81b0-ef4fedc44fcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:31.253395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11968277-91f3-4952-81b0-ef4fedc44fcb admin admin] PATH_INFO: `/v3/projects/04f55434b5cd4bacbb4cd77b47d36015` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:31.257252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11968277-91f3-4952-81b0-ef4fedc44fcb admin admin] RBAC: Authorizing `identity:delete_project(project_id=04f55434b5cd4bacbb4cd77b47d36015)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:31.261264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11968277-91f3-4952-81b0-ef4fedc44fcb admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:31.294201 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 879/1750] 10.222.0.49 () {58 vars in 1015 bytes} [Fri Sep 23 14:02:31 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:31.296268 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc672c0-85b7-4792-ac79-b7b908123a48 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:31.309101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-3dc672c0-85b7-4792-ac79-b7b908123a48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:31.343292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc672c0-85b7-4792-ac79-b7b908123a48 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:31.349604 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b5f7b0-e5fd-4554-b70d-142e80ad6ff0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:31.350746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b5f7b0-e5fd-4554-b70d-142e80ad6ff0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:31.351302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b5f7b0-e5fd-4554-b70d-142e80ad6ff0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:31.351936 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b5f7b0-e5fd-4554-b70d-142e80ad6ff0 admin admin] PATH_INFO: `/v3/projects/59c7d00de35544e1ad06e8b4f290e6c0/users/ac803da0c82141a78b9093c1e360dafe/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:31.354425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b5f7b0-e5fd-4554-b70d-142e80ad6ff0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=59c7d00de35544e1ad06e8b4f290e6c0, user_id=ac803da0c82141a78b9093c1e360dafe, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:31.357522 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b5f7b0-e5fd-4554-b70d-142e80ad6ff0 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:31.382306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 880/1751] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:31 2022] PUT /identity/v3/projects/59c7d00de35544e1ad06e8b4f290e6c0/users/ac803da0c82141a78b9093c1e360dafe/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:31.382306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d17fe65-4448-4b12-b4d6-9427b5756db6 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:31.382306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d17fe65-4448-4b12-b4d6-9427b5756db6 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:31.382306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d17fe65-4448-4b12-b4d6-9427b5756db6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:31.454626 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-1d17fe65-4448-4b12-b4d6-9427b5756db6 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:31.455655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-11968277-91f3-4952-81b0-ef4fedc44fcb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '04f55434b5cd4bacbb4cd77b47d36015', 'request_id': 'req-11968277-91f3-4952-81b0-ef4fedc44fcb'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:31.456112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-1d17fe65-4448-4b12-b4d6-9427b5756db6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:31.663535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 881/1752] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:31.663535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6a66c6-0daf-4ef2-a33e-c522427936c2 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:31.663535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6a66c6-0daf-4ef2-a33e-c522427936c2 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:31.663535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6a66c6-0daf-4ef2-a33e-c522427936c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:31.663535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-7d6a66c6-0daf-4ef2-a33e-c522427936c2 None None] MFA Rules not processed for user `ac803da0c82141a78b9093c1e360dafe`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:31.663535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-7d6a66c6-0daf-4ef2-a33e-c522427936c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:31.692310 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 882/1753] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:31 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:31.692310 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ee1b38-bbe5-41fd-8d5b-de5683fcd7b1 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:31.692310 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ee1b38-bbe5-41fd-8d5b-de5683fcd7b1 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:31.692310 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ee1b38-bbe5-41fd-8d5b-de5683fcd7b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:31.745429 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-a5ee1b38-bbe5-41fd-8d5b-de5683fcd7b1 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:31.745429 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a5ee1b38-bbe5-41fd-8d5b-de5683fcd7b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:31.745429 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 883/1754] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:31.745429 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ee1b38-bbe5-41fd-8d5b-de5683fcd7b1 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:31.745429 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a5ee1b38-bbe5-41fd-8d5b-de5683fcd7b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:31.763513 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-11968277-91f3-4952-81b0-ef4fedc44fcb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '04f55434b5cd4bacbb4cd77b47d36015', 'request_id': 'req-11968277-91f3-4952-81b0-ef4fedc44fcb'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:31.770765 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-11968277-91f3-4952-81b0-ef4fedc44fcb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '04f55434b5cd4bacbb4cd77b47d36015', 'request_id': 'req-11968277-91f3-4952-81b0-ef4fedc44fcb'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:31.815355 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-11968277-91f3-4952-81b0-ef4fedc44fcb admin admin] The token cache is being invalidate because project 04f55434b5cd4bacbb4cd77b47d36015 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:31.815355 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-11968277-91f3-4952-81b0-ef4fedc44fcb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:31.815355 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-11968277-91f3-4952-81b0-ef4fedc44fcb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:31.815355 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 872/1755] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:31 2022] DELETE /identity/v3/projects/04f55434b5cd4bacbb4cd77b47d36015 => generated 0 bytes in 600 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:31.815355 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11968277-91f3-4952-81b0-ef4fedc44fcb admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:31.816041 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-11968277-91f3-4952-81b0-ef4fedc44fcb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:31.823690 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ee1b38-bbe5-41fd-8d5b-de5683fcd7b1 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:31.828044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b587ef-5b03-4a93-b9b9-44be6dd9236d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:31.828665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b587ef-5b03-4a93-b9b9-44be6dd9236d service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:31.828943 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b587ef-5b03-4a93-b9b9-44be6dd9236d service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:31.829218 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72b587ef-5b03-4a93-b9b9-44be6dd9236d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:31.845867 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-72b587ef-5b03-4a93-b9b9-44be6dd9236d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:31.918520 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11968277-91f3-4952-81b0-ef4fedc44fcb admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:31.919180 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b587ef-5b03-4a93-b9b9-44be6dd9236d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:31.919416 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb407c5-af31-410c-99f9-b53675e41942 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:31.919416 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb407c5-af31-410c-99f9-b53675e41942 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:31.919416 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb407c5-af31-410c-99f9-b53675e41942 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:31.919622 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72b587ef-5b03-4a93-b9b9-44be6dd9236d service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:31.919698 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb407c5-af31-410c-99f9-b53675e41942 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:31.919698 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ceb407c5-af31-410c-99f9-b53675e41942 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:31.919826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 884/1756] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:31 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:31.919826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b587ef-5b03-4a93-b9b9-44be6dd9236d service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:31.919976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb407c5-af31-410c-99f9-b53675e41942 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:31.919976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb407c5-af31-410c-99f9-b53675e41942 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:31.920133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-72b587ef-5b03-4a93-b9b9-44be6dd9236d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:31.928108 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 873/1757] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:31 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:31.929361 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb407c5-af31-410c-99f9-b53675e41942 service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:31.930824 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ceb407c5-af31-410c-99f9-b53675e41942 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:31.949592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb407c5-af31-410c-99f9-b53675e41942 service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:31.966387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72b587ef-5b03-4a93-b9b9-44be6dd9236d service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:31.975784 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5664f75c-6f55-4128-bd2e-e3db864d79d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:31.976585 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5664f75c-6f55-4128-bd2e-e3db864d79d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:31.976981 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5664f75c-6f55-4128-bd2e-e3db864d79d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:31.977353 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5664f75c-6f55-4128-bd2e-e3db864d79d8 admin admin] PATH_INFO: `/v3/users/3432bae1c2ed4f6fa40333efc65f971c` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:31.980711 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5664f75c-6f55-4128-bd2e-e3db864d79d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3432bae1c2ed4f6fa40333efc65f971c)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:31.982869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5664f75c-6f55-4128-bd2e-e3db864d79d8 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:31.984929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02462e18-d531-4b14-87e9-aced42887002 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:31.985599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02462e18-d531-4b14-87e9-aced42887002 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:31.985877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02462e18-d531-4b14-87e9-aced42887002 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:31.986174 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02462e18-d531-4b14-87e9-aced42887002 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:31.986927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02462e18-d531-4b14-87e9-aced42887002 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:31.987236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02462e18-d531-4b14-87e9-aced42887002 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:31.988989 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02462e18-d531-4b14-87e9-aced42887002 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:32.017302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 885/1758] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:02:32.018158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02462e18-d531-4b14-87e9-aced42887002 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:32.026954 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-02462e18-d531-4b14-87e9-aced42887002 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:32.061290 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02462e18-d531-4b14-87e9-aced42887002 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:32.072610 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f04c552-de5f-43db-80ff-8f4ebdac4aa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:32.081201 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f04c552-de5f-43db-80ff-8f4ebdac4aa3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:32.081201 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f04c552-de5f-43db-80ff-8f4ebdac4aa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:32.081201 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f04c552-de5f-43db-80ff-8f4ebdac4aa3 admin admin] PATH_INFO: `/v3/projects/addcb88552534c7589ab2d0093d65ab7` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:32.081201 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f04c552-de5f-43db-80ff-8f4ebdac4aa3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=addcb88552534c7589ab2d0093d65ab7)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:32.081201 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f04c552-de5f-43db-80ff-8f4ebdac4aa3 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:32.333880 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6f04c552-de5f-43db-80ff-8f4ebdac4aa3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'addcb88552534c7589ab2d0093d65ab7', 'request_id': 'req-6f04c552-de5f-43db-80ff-8f4ebdac4aa3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:32.346282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5664f75c-6f55-4128-bd2e-e3db864d79d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3432bae1c2ed4f6fa40333efc65f971c'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:32.449046 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6f04c552-de5f-43db-80ff-8f4ebdac4aa3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'addcb88552534c7589ab2d0093d65ab7', 'request_id': 'req-6f04c552-de5f-43db-80ff-8f4ebdac4aa3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:32.471039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6f04c552-de5f-43db-80ff-8f4ebdac4aa3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'addcb88552534c7589ab2d0093d65ab7', 'request_id': 'req-6f04c552-de5f-43db-80ff-8f4ebdac4aa3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:32.489905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6f04c552-de5f-43db-80ff-8f4ebdac4aa3 admin admin] The token cache is being invalidate because project addcb88552534c7589ab2d0093d65ab7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:32.490194 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6f04c552-de5f-43db-80ff-8f4ebdac4aa3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:32.491037 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6f04c552-de5f-43db-80ff-8f4ebdac4aa3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:32.498569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 886/1759] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:32 2022] DELETE /identity/v3/projects/addcb88552534c7589ab2d0093d65ab7 => generated 0 bytes in 475 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:32.499667 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f04c552-de5f-43db-80ff-8f4ebdac4aa3 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:32.501232 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-6f04c552-de5f-43db-80ff-8f4ebdac4aa3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:32.573794 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5664f75c-6f55-4128-bd2e-e3db864d79d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3432bae1c2ed4f6fa40333efc65f971c'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:32.579464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5664f75c-6f55-4128-bd2e-e3db864d79d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3432bae1c2ed4f6fa40333efc65f971c'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:32.598364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f04c552-de5f-43db-80ff-8f4ebdac4aa3 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:32.616737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b166c9-3dd6-4fd9-a576-d8e191855d5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:32.617494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b166c9-3dd6-4fd9-a576-d8e191855d5b admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:32.617806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b166c9-3dd6-4fd9-a576-d8e191855d5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:32.618253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b166c9-3dd6-4fd9-a576-d8e191855d5b admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:32.618988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b166c9-3dd6-4fd9-a576-d8e191855d5b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:32.621356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b166c9-3dd6-4fd9-a576-d8e191855d5b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:32.631907 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5664f75c-6f55-4128-bd2e-e3db864d79d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3432bae1c2ed4f6fa40333efc65f971c'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:32.663004 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5664f75c-6f55-4128-bd2e-e3db864d79d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3432bae1c2ed4f6fa40333efc65f971c'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:32.680755 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 874/1760] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:31 2022] DELETE /identity/v3/users/3432bae1c2ed4f6fa40333efc65f971c => generated 0 bytes in 752 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:32.684877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5664f75c-6f55-4128-bd2e-e3db864d79d8 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:32.684877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-5664f75c-6f55-4128-bd2e-e3db864d79d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:32.773994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 887/1761] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:32 2022] POST /identity/v3/projects => generated 373 bytes in 275 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:32.776539 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659cd235-bf4f-46b2-95ae-e74fc57ce445 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:32.776998 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659cd235-bf4f-46b2-95ae-e74fc57ce445 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:32.777457 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659cd235-bf4f-46b2-95ae-e74fc57ce445 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:32.819334 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-659cd235-bf4f-46b2-95ae-e74fc57ce445 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:32.820962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-659cd235-bf4f-46b2-95ae-e74fc57ce445 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:32.883501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5664f75c-6f55-4128-bd2e-e3db864d79d8 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:32.954462 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 888/1762] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:32.954462 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659cd235-bf4f-46b2-95ae-e74fc57ce445 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:32.954462 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-659cd235-bf4f-46b2-95ae-e74fc57ce445 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:32.969648 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659cd235-bf4f-46b2-95ae-e74fc57ce445 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:32.979832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43629e1f-6509-461d-b64e-1c7594345093 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:32.982188 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43629e1f-6509-461d-b64e-1c7594345093 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:32.982576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43629e1f-6509-461d-b64e-1c7594345093 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:32.982852 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43629e1f-6509-461d-b64e-1c7594345093 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:32.983855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43629e1f-6509-461d-b64e-1c7594345093 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:32.988527 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a21485-9298-4fe1-977a-ca73df1ea23b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:32.992101 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a21485-9298-4fe1-977a-ca73df1ea23b service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:32.992101 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a21485-9298-4fe1-977a-ca73df1ea23b service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:32.992101 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a21485-9298-4fe1-977a-ca73df1ea23b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:32.992890 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43629e1f-6509-461d-b64e-1c7594345093 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:32.996951 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f3a21485-9298-4fe1-977a-ca73df1ea23b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:33.030083 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:33.126399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 889/1763] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:32 2022] POST /identity/v3/users => generated 287 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:33.128466 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4b88be-34d0-4b69-9476-8eacdedfabc7 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:33.129720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4b88be-34d0-4b69-9476-8eacdedfabc7 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:33.130913 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4b88be-34d0-4b69-9476-8eacdedfabc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:33.150092 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a21485-9298-4fe1-977a-ca73df1ea23b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:33.151683 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a21485-9298-4fe1-977a-ca73df1ea23b service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:33.194923 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 875/1764] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:32 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 513 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:33.196648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a21485-9298-4fe1-977a-ca73df1ea23b service cinder] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:33.203905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-2d4b88be-34d0-4b69-9476-8eacdedfabc7 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:33.213558 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-2d4b88be-34d0-4b69-9476-8eacdedfabc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:33.235658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a21485-9298-4fe1-977a-ca73df1ea23b service cinder] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:33.243191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 890/1765] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:33.247320 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4b88be-34d0-4b69-9476-8eacdedfabc7 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:33.264352 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e013db59-4d4d-4513-b577-07d31a2169d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:33.267208 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e013db59-4d4d-4513-b577-07d31a2169d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:33.267724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e013db59-4d4d-4513-b577-07d31a2169d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:33.272087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e013db59-4d4d-4513-b577-07d31a2169d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:33.274783 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e013db59-4d4d-4513-b577-07d31a2169d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:33.284432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e013db59-4d4d-4513-b577-07d31a2169d2 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:33.287960 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4b88be-34d0-4b69-9476-8eacdedfabc7 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:33.308078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a10aa97-0866-42ca-bbe9-818163283c71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:33.308910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a10aa97-0866-42ca-bbe9-818163283c71 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:33.309317 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a10aa97-0866-42ca-bbe9-818163283c71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:33.312929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a10aa97-0866-42ca-bbe9-818163283c71 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:33.316452 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:33.316770 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 876/1766] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:33 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:33.317653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a10aa97-0866-42ca-bbe9-818163283c71 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:33.318225 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a10aa97-0866-42ca-bbe9-818163283c71 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:33.320251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a10aa97-0866-42ca-bbe9-818163283c71 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:33.323829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e013db59-4d4d-4513-b577-07d31a2169d2 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:33.354883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e013db59-4d4d-4513-b577-07d31a2169d2 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:33.355552 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 891/1767] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:02:33.357856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a10aa97-0866-42ca-bbe9-818163283c71 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:33.362991 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd9f53b-a409-4535-8246-4546654f6df2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:33.364062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd9f53b-a409-4535-8246-4546654f6df2 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:33.364592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd9f53b-a409-4535-8246-4546654f6df2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:33.365051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd9f53b-a409-4535-8246-4546654f6df2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:33.366109 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd9f53b-a409-4535-8246-4546654f6df2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:33.369715 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd9f53b-a409-4535-8246-4546654f6df2 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:33.389301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 877/1768] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:33 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:33.394275 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd9f53b-a409-4535-8246-4546654f6df2 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:33.409019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a10aa97-0866-42ca-bbe9-818163283c71 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:33.409129 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd9f53b-a409-4535-8246-4546654f6df2 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:33.419436 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c20d1c-4638-4643-81ea-eef3af625cb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:33.420338 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c20d1c-4638-4643-81ea-eef3af625cb2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:33.420683 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c20d1c-4638-4643-81ea-eef3af625cb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:33.421221 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c20d1c-4638-4643-81ea-eef3af625cb2 admin admin] PATH_INFO: `/v3/projects/82a6b6347cee452782d71bac772d0e3f/users/2f09d40387cd4e1d844df434a197da99/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:33.431955 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b767b24-ac5f-469d-b15f-8310ebed13c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:33.432872 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b767b24-ac5f-469d-b15f-8310ebed13c0 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:33.433173 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b767b24-ac5f-469d-b15f-8310ebed13c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:33.433455 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b767b24-ac5f-469d-b15f-8310ebed13c0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:33.434120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b767b24-ac5f-469d-b15f-8310ebed13c0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:33.436166 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b767b24-ac5f-469d-b15f-8310ebed13c0 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:33.464599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c20d1c-4638-4643-81ea-eef3af625cb2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=82a6b6347cee452782d71bac772d0e3f, user_id=2f09d40387cd4e1d844df434a197da99, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:33.483887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c20d1c-4638-4643-81ea-eef3af625cb2 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:33.589253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 878/1769] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:33 2022] PUT /identity/v3/projects/82a6b6347cee452782d71bac772d0e3f/users/2f09d40387cd4e1d844df434a197da99/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:33.592145 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ca05fa-6f64-4de8-84ea-1f078b0033f0 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:33.592437 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ca05fa-6f64-4de8-84ea-1f078b0033f0 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:33.592862 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ca05fa-6f64-4de8-84ea-1f078b0033f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:33.594910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 892/1770] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:33 2022] POST /identity/v3/projects => generated 355 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:33.599357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b767b24-ac5f-469d-b15f-8310ebed13c0 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:33.694076 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-b9ca05fa-6f64-4de8-84ea-1f078b0033f0 None None] MFA Rules not processed for user `2f09d40387cd4e1d844df434a197da99`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:33.695659 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-b9ca05fa-6f64-4de8-84ea-1f078b0033f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:33.781002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b767b24-ac5f-469d-b15f-8310ebed13c0 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:33.792133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39ae36d-e863-4479-ac56-1b82a6578514 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:33.795496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39ae36d-e863-4479-ac56-1b82a6578514 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:33.795496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39ae36d-e863-4479-ac56-1b82a6578514 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:33.795496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39ae36d-e863-4479-ac56-1b82a6578514 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:33.795496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39ae36d-e863-4479-ac56-1b82a6578514 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:33.802590 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c39ae36d-e863-4479-ac56-1b82a6578514 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:33.920060 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 879/1771] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:33 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 329 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:02:33.924685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ca05fa-6f64-4de8-84ea-1f078b0033f0 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:33.944510 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ca05fa-6f64-4de8-84ea-1f078b0033f0 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:33.949472 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 893/1772] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:33 2022] POST /identity/v3/users => generated 278 bytes in 351 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:02:33.952758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b429ac-4e10-4b6d-bf0c-007c0ec090d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:33.952758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b429ac-4e10-4b6d-bf0c-007c0ec090d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:33.952981 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b429ac-4e10-4b6d-bf0c-007c0ec090d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:33.953256 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b429ac-4e10-4b6d-bf0c-007c0ec090d7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:33.953920 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b429ac-4e10-4b6d-bf0c-007c0ec090d7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:33.954451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39ae36d-e863-4479-ac56-1b82a6578514 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:33.956165 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b429ac-4e10-4b6d-bf0c-007c0ec090d7 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:33.978005 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39ae36d-e863-4479-ac56-1b82a6578514 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:33.992680 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d999603c-043d-4e83-943e-6ee5556ce244 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:33.993530 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d999603c-043d-4e83-943e-6ee5556ce244 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:33.994277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d999603c-043d-4e83-943e-6ee5556ce244 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:33.994773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d999603c-043d-4e83-943e-6ee5556ce244 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:33.999369 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d999603c-043d-4e83-943e-6ee5556ce244 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:34.003094 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d999603c-043d-4e83-943e-6ee5556ce244 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:34.032202 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 894/1773] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:33 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:34.034114 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d999603c-043d-4e83-943e-6ee5556ce244 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:34.055567 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d999603c-043d-4e83-943e-6ee5556ce244 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:34.064931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9abb2c7-2b54-42d9-bb2a-5b485ca3b8d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:34.069499 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9abb2c7-2b54-42d9-bb2a-5b485ca3b8d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:34.077160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9abb2c7-2b54-42d9-bb2a-5b485ca3b8d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:34.077160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9abb2c7-2b54-42d9-bb2a-5b485ca3b8d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:34.077160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9abb2c7-2b54-42d9-bb2a-5b485ca3b8d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:34.077160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9abb2c7-2b54-42d9-bb2a-5b485ca3b8d9 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:34.093331 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 895/1774] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:34 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:34.095234 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9abb2c7-2b54-42d9-bb2a-5b485ca3b8d9 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:34.097229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d9abb2c7-2b54-42d9-bb2a-5b485ca3b8d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:34.124485 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 880/1775] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:33 2022] POST /identity/v3/projects => generated 371 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:34.125590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b429ac-4e10-4b6d-bf0c-007c0ec090d7 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:34.155346 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9abb2c7-2b54-42d9-bb2a-5b485ca3b8d9 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:34.320877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b429ac-4e10-4b6d-bf0c-007c0ec090d7 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:34.332749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e60aa07-5759-485e-b513-82ec420a2559 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:34.335259 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e60aa07-5759-485e-b513-82ec420a2559 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:34.336284 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e60aa07-5759-485e-b513-82ec420a2559 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:34.336592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e60aa07-5759-485e-b513-82ec420a2559 admin admin] PATH_INFO: `/v3/projects/4e5c2dfbf6e8451aa06c3193fc850a22/users/3ca42b8753dc40f7a9127f6582298770/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:34.339080 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3598a6e9-9031-497b-b00e-6745fb758c6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:34.339858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3598a6e9-9031-497b-b00e-6745fb758c6d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:34.341624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3598a6e9-9031-497b-b00e-6745fb758c6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:34.341924 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3598a6e9-9031-497b-b00e-6745fb758c6d admin admin] PATH_INFO: `/v3/projects/da4abb67ad4d4464b74e5ccbbd09b426` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:34.343678 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3598a6e9-9031-497b-b00e-6745fb758c6d admin admin] RBAC: Authorizing `identity:delete_project(project_id=da4abb67ad4d4464b74e5ccbbd09b426)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:34.371467 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3598a6e9-9031-497b-b00e-6745fb758c6d admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:34.415446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e60aa07-5759-485e-b513-82ec420a2559 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4e5c2dfbf6e8451aa06c3193fc850a22, user_id=3ca42b8753dc40f7a9127f6582298770, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:34.429967 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e60aa07-5759-485e-b513-82ec420a2559 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:34.555456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 881/1776] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:34 2022] PUT /identity/v3/projects/4e5c2dfbf6e8451aa06c3193fc850a22/users/3ca42b8753dc40f7a9127f6582298770/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 430 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:34.556551 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e60aa07-5759-485e-b513-82ec420a2559 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:34.667446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-3598a6e9-9031-497b-b00e-6745fb758c6d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'da4abb67ad4d4464b74e5ccbbd09b426', 'request_id': 'req-3598a6e9-9031-497b-b00e-6745fb758c6d'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:34.794773 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e60aa07-5759-485e-b513-82ec420a2559 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:34.807253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-3598a6e9-9031-497b-b00e-6745fb758c6d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'da4abb67ad4d4464b74e5ccbbd09b426', 'request_id': 'req-3598a6e9-9031-497b-b00e-6745fb758c6d'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:34.819474 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-3598a6e9-9031-497b-b00e-6745fb758c6d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'da4abb67ad4d4464b74e5ccbbd09b426', 'request_id': 'req-3598a6e9-9031-497b-b00e-6745fb758c6d'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:34.842667 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-3598a6e9-9031-497b-b00e-6745fb758c6d admin admin] The token cache is being invalidate because project da4abb67ad4d4464b74e5ccbbd09b426 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:34.845910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-3598a6e9-9031-497b-b00e-6745fb758c6d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:34.845910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-3598a6e9-9031-497b-b00e-6745fb758c6d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:34.855499 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 896/1777] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:34 2022] DELETE /identity/v3/projects/da4abb67ad4d4464b74e5ccbbd09b426 => generated 0 bytes in 762 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:34.858955 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d06d19-8997-4ffd-a0e0-7f2a80ce8bab None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:34.860618 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d06d19-8997-4ffd-a0e0-7f2a80ce8bab None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:34.861304 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d06d19-8997-4ffd-a0e0-7f2a80ce8bab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:34.925207 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97c45b6-4457-4a22-becf-96765a360678 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:34.925906 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97c45b6-4457-4a22-becf-96765a360678 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:34.926125 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97c45b6-4457-4a22-becf-96765a360678 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:34.931711 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97c45b6-4457-4a22-becf-96765a360678 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:34.931711 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97c45b6-4457-4a22-becf-96765a360678 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:34.936115 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-00d06d19-8997-4ffd-a0e0-7f2a80ce8bab None None] MFA Rules not processed for user `3ca42b8753dc40f7a9127f6582298770`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:34.938090 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-00d06d19-8997-4ffd-a0e0-7f2a80ce8bab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:34.940601 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e97c45b6-4457-4a22-becf-96765a360678 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:34.977182 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:35.048987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 882/1778] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:34 2022] POST /identity/v3/users => generated 292 bytes in 492 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:35.050587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5d4382-714a-4c4f-b1f6-bf51d12d24a4 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:35.050888 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5d4382-714a-4c4f-b1f6-bf51d12d24a4 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:35.051167 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5d4382-714a-4c4f-b1f6-bf51d12d24a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:35.064787 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 897/1779] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:34 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:35.065764 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d06d19-8997-4ffd-a0e0-7f2a80ce8bab None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:35.067256 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-00d06d19-8997-4ffd-a0e0-7f2a80ce8bab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:35.086103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d06d19-8997-4ffd-a0e0-7f2a80ce8bab None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:35.096300 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-cf5d4382-714a-4c4f-b1f6-bf51d12d24a4 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:35.097746 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-cf5d4382-714a-4c4f-b1f6-bf51d12d24a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:35.106634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de7f049-4e0e-48e9-abb5-7e652d7e1bd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:35.107657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de7f049-4e0e-48e9-abb5-7e652d7e1bd5 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:35.108631 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de7f049-4e0e-48e9-abb5-7e652d7e1bd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:35.109902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de7f049-4e0e-48e9-abb5-7e652d7e1bd5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:35.112913 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de7f049-4e0e-48e9-abb5-7e652d7e1bd5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:35.117512 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 883/1780] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:35.119185 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5d4382-714a-4c4f-b1f6-bf51d12d24a4 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:35.119866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de7f049-4e0e-48e9-abb5-7e652d7e1bd5 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:35.123121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-cf5d4382-714a-4c4f-b1f6-bf51d12d24a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:35.148184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 898/1781] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:35 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:35.148184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1a82c0-f5ba-4e8a-a395-521f15aa57a1 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:35.149313 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1a82c0-f5ba-4e8a-a395-521f15aa57a1 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:35.149313 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1a82c0-f5ba-4e8a-a395-521f15aa57a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:35.157051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:35.159596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5d4382-714a-4c4f-b1f6-bf51d12d24a4 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:35.168385 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc71731f-a335-4549-8603-6161df114139 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:35.169301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc71731f-a335-4549-8603-6161df114139 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:35.169656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc71731f-a335-4549-8603-6161df114139 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:35.169831 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc71731f-a335-4549-8603-6161df114139 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:35.170503 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc71731f-a335-4549-8603-6161df114139 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:35.173337 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc71731f-a335-4549-8603-6161df114139 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:35.200597 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-9d1a82c0-f5ba-4e8a-a395-521f15aa57a1 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:35.201934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-9d1a82c0-f5ba-4e8a-a395-521f15aa57a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:35.227664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 899/1782] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:35.231062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2915c62e-39c9-46c3-8bd2-4b1e52e073a1 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:35.231452 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2915c62e-39c9-46c3-8bd2-4b1e52e073a1 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:35.231885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2915c62e-39c9-46c3-8bd2-4b1e52e073a1 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:35.233741 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 900/1783] 10.222.0.62 () {54 vars in 921 bytes} [Fri Sep 23 14:02:35 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:02:35.234703 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2915c62e-39c9-46c3-8bd2-4b1e52e073a1 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:35.253960 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2915c62e-39c9-46c3-8bd2-4b1e52e073a1 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:35.260631 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b02c098-a2f1-4d13-810f-ef437571ea5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:35.261476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b02c098-a2f1-4d13-810f-ef437571ea5e admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:35.261766 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b02c098-a2f1-4d13-810f-ef437571ea5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:35.262037 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b02c098-a2f1-4d13-810f-ef437571ea5e admin admin] PATH_INFO: `/v3/projects/fc72405ff8bc4e3bb977083d341f1f5c/users/1a5c2019a1cb487f84d706365f30ec8f/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:35.278955 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b02c098-a2f1-4d13-810f-ef437571ea5e admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc72405ff8bc4e3bb977083d341f1f5c, user_id=1a5c2019a1cb487f84d706365f30ec8f, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:35.281857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b02c098-a2f1-4d13-810f-ef437571ea5e admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:35.286455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 884/1784] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:35 2022] POST /identity/v3/projects => generated 357 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:35.286888 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc71731f-a335-4549-8603-6161df114139 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:35.366524 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc71731f-a335-4549-8603-6161df114139 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:35.371542 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be4c98a-4ff5-44a2-b61e-811d264403b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:35.372166 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be4c98a-4ff5-44a2-b61e-811d264403b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:35.373389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be4c98a-4ff5-44a2-b61e-811d264403b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:35.373389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be4c98a-4ff5-44a2-b61e-811d264403b6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:35.373389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be4c98a-4ff5-44a2-b61e-811d264403b6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:35.373688 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be4c98a-4ff5-44a2-b61e-811d264403b6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:35.375318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be4c98a-4ff5-44a2-b61e-811d264403b6 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:35.380831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 901/1785] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:35 2022] PUT /identity/v3/projects/fc72405ff8bc4e3bb977083d341f1f5c/users/1a5c2019a1cb487f84d706365f30ec8f/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:35.382495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79892e1-632b-42cd-94ef-ca0f05b8d741 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:35.382780 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79892e1-632b-42cd-94ef-ca0f05b8d741 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:35.383077 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79892e1-632b-42cd-94ef-ca0f05b8d741 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:35.393970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 885/1786] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:02:35.393970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be4c98a-4ff5-44a2-b61e-811d264403b6 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:35.403023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-f79892e1-632b-42cd-94ef-ca0f05b8d741 None None] MFA Rules not processed for user `f01ed94dbb384a178ad0a45a6f3d8c17`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:35.404228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f79892e1-632b-42cd-94ef-ca0f05b8d741 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:35.404494 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be4c98a-4ff5-44a2-b61e-811d264403b6 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:35.498022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f952fb-16a6-4a6b-8181-fd1cb07c36c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:35.502842 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f952fb-16a6-4a6b-8181-fd1cb07c36c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:35.502842 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f952fb-16a6-4a6b-8181-fd1cb07c36c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:35.502842 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f952fb-16a6-4a6b-8181-fd1cb07c36c6 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:35.503118 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 902/1787] 10.222.0.62 () {58 vars in 1018 bytes} [Fri Sep 23 14:02:35 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:35.503118 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79892e1-632b-42cd-94ef-ca0f05b8d741 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:35.503693 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f952fb-16a6-4a6b-8181-fd1cb07c36c6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:35.507487 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f952fb-16a6-4a6b-8181-fd1cb07c36c6 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:35.526489 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79892e1-632b-42cd-94ef-ca0f05b8d741 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:35.534213 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d2f344-c4f8-49a5-a1e2-dc3b0b7baae6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:35.534874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d2f344-c4f8-49a5-a1e2-dc3b0b7baae6 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:35.535151 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d2f344-c4f8-49a5-a1e2-dc3b0b7baae6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:35.535422 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d2f344-c4f8-49a5-a1e2-dc3b0b7baae6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:35.536610 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d2f344-c4f8-49a5-a1e2-dc3b0b7baae6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:35.538517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d2f344-c4f8-49a5-a1e2-dc3b0b7baae6 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:35.549269 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 903/1788] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:35 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:35.551260 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d2f344-c4f8-49a5-a1e2-dc3b0b7baae6 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:35.562920 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d2f344-c4f8-49a5-a1e2-dc3b0b7baae6 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:35.571189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a952bcb-579b-49bd-b248-b8710ff54508 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:35.571189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a952bcb-579b-49bd-b248-b8710ff54508 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:35.571189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a952bcb-579b-49bd-b248-b8710ff54508 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:35.571189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a952bcb-579b-49bd-b248-b8710ff54508 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:35.571547 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a952bcb-579b-49bd-b248-b8710ff54508 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:35.572112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a952bcb-579b-49bd-b248-b8710ff54508 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:35.717042 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 886/1789] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:35 2022] POST /identity/v3/users => generated 285 bytes in 324 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:35.717875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f952fb-16a6-4a6b-8181-fd1cb07c36c6 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:35.719588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a5f952fb-16a6-4a6b-8181-fd1cb07c36c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:35.740977 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 904/1790] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:35 2022] POST /identity/v3/projects => generated 365 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:35.741111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f952fb-16a6-4a6b-8181-fd1cb07c36c6 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:35.741204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a952bcb-579b-49bd-b248-b8710ff54508 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:35.810604 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a4e142-2a7c-4f65-a2ca-61974b26eebc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:35.811354 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a4e142-2a7c-4f65-a2ca-61974b26eebc service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:35.811663 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a4e142-2a7c-4f65-a2ca-61974b26eebc service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:35.811941 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a4e142-2a7c-4f65-a2ca-61974b26eebc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:35.812142 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a952bcb-579b-49bd-b248-b8710ff54508 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:35.817491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76b6dd8-fe76-4930-83f7-ed33e824cfc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:35.818261 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76b6dd8-fe76-4930-83f7-ed33e824cfc2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:35.818576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76b6dd8-fe76-4930-83f7-ed33e824cfc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:35.818977 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76b6dd8-fe76-4930-83f7-ed33e824cfc2 admin admin] PATH_INFO: `/v3/projects/fc72405ff8bc4e3bb977083d341f1f5c/users/1a5c2019a1cb487f84d706365f30ec8f/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:35.822286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76b6dd8-fe76-4930-83f7-ed33e824cfc2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc72405ff8bc4e3bb977083d341f1f5c, user_id=1a5c2019a1cb487f84d706365f30ec8f, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:35.823837 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76b6dd8-fe76-4930-83f7-ed33e824cfc2 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:35.835323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 905/1791] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:35 2022] PUT /identity/v3/projects/fc72405ff8bc4e3bb977083d341f1f5c/users/1a5c2019a1cb487f84d706365f30ec8f/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:35.836177 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76b6dd8-fe76-4930-83f7-ed33e824cfc2 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:35.854832 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a4e142-2a7c-4f65-a2ca-61974b26eebc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:35.854832 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a4e142-2a7c-4f65-a2ca-61974b26eebc service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:35.862803 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 887/1792] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:35.863526 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a4e142-2a7c-4f65-a2ca-61974b26eebc service nova] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:35.875128 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a4e142-2a7c-4f65-a2ca-61974b26eebc service nova] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:35.930996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76b6dd8-fe76-4930-83f7-ed33e824cfc2 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:35.939498 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7a7524-bb93-4c71-adaa-a1411d70f9bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:35.940180 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7a7524-bb93-4c71-adaa-a1411d70f9bb admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:35.940621 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7a7524-bb93-4c71-adaa-a1411d70f9bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:35.940898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7a7524-bb93-4c71-adaa-a1411d70f9bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:35.941719 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7a7524-bb93-4c71-adaa-a1411d70f9bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:35.944052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7a7524-bb93-4c71-adaa-a1411d70f9bb admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:35.951114 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff27829-2ac6-4289-837d-079b86221279 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:35.953456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff27829-2ac6-4289-837d-079b86221279 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:35.953456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff27829-2ac6-4289-837d-079b86221279 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:35.953456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff27829-2ac6-4289-837d-079b86221279 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:35.953456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff27829-2ac6-4289-837d-079b86221279 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:35.955026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff27829-2ac6-4289-837d-079b86221279 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:35.960014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 906/1793] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:35 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:35.964231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82b38a3-1d7a-490b-aa06-8b4398bb13fc None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:35.964496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82b38a3-1d7a-490b-aa06-8b4398bb13fc None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:35.964926 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82b38a3-1d7a-490b-aa06-8b4398bb13fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:36.006517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c82b38a3-1d7a-490b-aa06-8b4398bb13fc None None] MFA Rules not processed for user `1a5c2019a1cb487f84d706365f30ec8f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:36.011381 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c82b38a3-1d7a-490b-aa06-8b4398bb13fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:36.063720 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 888/1794] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:35 2022] POST /identity/v3/users => generated 283 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:36.065015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff27829-2ac6-4289-837d-079b86221279 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:36.084882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff27829-2ac6-4289-837d-079b86221279 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:36.092987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828f4c21-c0cf-4682-a5e9-f92ac78de30c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:36.093663 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828f4c21-c0cf-4682-a5e9-f92ac78de30c admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:36.093940 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828f4c21-c0cf-4682-a5e9-f92ac78de30c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:36.094208 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828f4c21-c0cf-4682-a5e9-f92ac78de30c admin admin] PATH_INFO: `/v3/projects/fce3614cf145410da0f9185326de2813/users/b5add60179a6493e9f58a42719fcc95a/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:36.110827 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828f4c21-c0cf-4682-a5e9-f92ac78de30c admin admin] RBAC: Authorizing `identity:create_grant(project_id=fce3614cf145410da0f9185326de2813, user_id=b5add60179a6493e9f58a42719fcc95a, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:36.114385 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828f4c21-c0cf-4682-a5e9-f92ac78de30c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:36.130527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 907/1795] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:35 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:36.135193 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82b38a3-1d7a-490b-aa06-8b4398bb13fc None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:36.142470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82b38a3-1d7a-490b-aa06-8b4398bb13fc None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:36.148344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805f7418-4d6b-4a43-8c14-afea48c0ca3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:36.152160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805f7418-4d6b-4a43-8c14-afea48c0ca3b admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:36.152160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805f7418-4d6b-4a43-8c14-afea48c0ca3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:36.152160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805f7418-4d6b-4a43-8c14-afea48c0ca3b admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:36.152160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805f7418-4d6b-4a43-8c14-afea48c0ca3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:36.152160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805f7418-4d6b-4a43-8c14-afea48c0ca3b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:36.162114 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 908/1796] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:36 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:36.162933 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805f7418-4d6b-4a43-8c14-afea48c0ca3b admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:36.164517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-805f7418-4d6b-4a43-8c14-afea48c0ca3b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:36.186373 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 889/1797] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:36 2022] PUT /identity/v3/projects/fce3614cf145410da0f9185326de2813/users/b5add60179a6493e9f58a42719fcc95a/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:36.190039 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828f4c21-c0cf-4682-a5e9-f92ac78de30c admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:36.225695 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805f7418-4d6b-4a43-8c14-afea48c0ca3b admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:36.262921 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828f4c21-c0cf-4682-a5e9-f92ac78de30c admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:36.267657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce11cc27-2a2c-4dd5-b249-61b107a366f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:36.268282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce11cc27-2a2c-4dd5-b249-61b107a366f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:36.268552 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce11cc27-2a2c-4dd5-b249-61b107a366f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:36.268827 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce11cc27-2a2c-4dd5-b249-61b107a366f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:36.269466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce11cc27-2a2c-4dd5-b249-61b107a366f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:36.272186 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce11cc27-2a2c-4dd5-b249-61b107a366f8 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:36.277748 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c082c012-5d3b-47b5-b4e1-36fd4f6c9e3e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:36.278543 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c082c012-5d3b-47b5-b4e1-36fd4f6c9e3e service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:36.278825 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c082c012-5d3b-47b5-b4e1-36fd4f6c9e3e service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:36.279096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c082c012-5d3b-47b5-b4e1-36fd4f6c9e3e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:36.282470 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 890/1798] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:36 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:36.287804 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce11cc27-2a2c-4dd5-b249-61b107a366f8 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:36.299395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce11cc27-2a2c-4dd5-b249-61b107a366f8 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:36.311093 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9013de51-cdde-44d4-a254-1f1528275fa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:36.311093 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9013de51-cdde-44d4-a254-1f1528275fa4 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:36.311093 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9013de51-cdde-44d4-a254-1f1528275fa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:36.311093 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9013de51-cdde-44d4-a254-1f1528275fa4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:36.311424 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9013de51-cdde-44d4-a254-1f1528275fa4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:36.311424 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9013de51-cdde-44d4-a254-1f1528275fa4 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:36.319944 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 891/1799] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:36 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:36.320746 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9013de51-cdde-44d4-a254-1f1528275fa4 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:36.330078 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9013de51-cdde-44d4-a254-1f1528275fa4 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:36.335961 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5015b6-3050-45d6-8dae-fe7b09e539c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:36.336611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5015b6-3050-45d6-8dae-fe7b09e539c0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:36.336921 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5015b6-3050-45d6-8dae-fe7b09e539c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:36.337267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5015b6-3050-45d6-8dae-fe7b09e539c0 admin admin] PATH_INFO: `/v3/projects/00ded4c4066a43b5933e10db7856ff10/users/6a17680ec574452a852cfdf9dc647170/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:36.346101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c082c012-5d3b-47b5-b4e1-36fd4f6c9e3e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:36.347569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c082c012-5d3b-47b5-b4e1-36fd4f6c9e3e service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:36.354268 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5015b6-3050-45d6-8dae-fe7b09e539c0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=00ded4c4066a43b5933e10db7856ff10, user_id=6a17680ec574452a852cfdf9dc647170, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:36.357132 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5015b6-3050-45d6-8dae-fe7b09e539c0 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:36.357742 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 909/1800] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:36 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:36.358544 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c082c012-5d3b-47b5-b4e1-36fd4f6c9e3e service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:36.369785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c082c012-5d3b-47b5-b4e1-36fd4f6c9e3e service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:36.374346 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d053cd5-2df5-48ef-ba88-b4377f19b6cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:36.375027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d053cd5-2df5-48ef-ba88-b4377f19b6cf admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:36.375302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d053cd5-2df5-48ef-ba88-b4377f19b6cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:36.375568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d053cd5-2df5-48ef-ba88-b4377f19b6cf admin admin] PATH_INFO: `/v3/projects/fce3614cf145410da0f9185326de2813/users/b5add60179a6493e9f58a42719fcc95a/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:36.377466 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d053cd5-2df5-48ef-ba88-b4377f19b6cf admin admin] RBAC: Authorizing `identity:create_grant(project_id=fce3614cf145410da0f9185326de2813, user_id=b5add60179a6493e9f58a42719fcc95a, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:36.380343 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d053cd5-2df5-48ef-ba88-b4377f19b6cf admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:36.390608 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 910/1801] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:36 2022] PUT /identity/v3/projects/fce3614cf145410da0f9185326de2813/users/b5add60179a6493e9f58a42719fcc95a/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:36.401530 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49b626f-364f-438d-94f3-b8f3984ac00f None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:36.401810 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49b626f-364f-438d-94f3-b8f3984ac00f None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:36.402082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49b626f-364f-438d-94f3-b8f3984ac00f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:36.431116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-e49b626f-364f-438d-94f3-b8f3984ac00f None None] MFA Rules not processed for user `b5add60179a6493e9f58a42719fcc95a`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:36.432359 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-e49b626f-364f-438d-94f3-b8f3984ac00f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:36.442992 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 892/1802] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:36 2022] PUT /identity/v3/projects/00ded4c4066a43b5933e10db7856ff10/users/6a17680ec574452a852cfdf9dc647170/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:36.461257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd54a233-6532-409e-9eed-f9e396a3cb62 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:36.461540 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd54a233-6532-409e-9eed-f9e396a3cb62 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:36.461818 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd54a233-6532-409e-9eed-f9e396a3cb62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:36.495990 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-fd54a233-6532-409e-9eed-f9e396a3cb62 None None] MFA Rules not processed for user `6a17680ec574452a852cfdf9dc647170`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:36.497240 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-fd54a233-6532-409e-9eed-f9e396a3cb62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:36.512485 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 911/1803] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:36 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:02:36.525111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49b626f-364f-438d-94f3-b8f3984ac00f None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:36.562894 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 893/1804] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:36 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:02:36.567344 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd54a233-6532-409e-9eed-f9e396a3cb62 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:36.584267 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49b626f-364f-438d-94f3-b8f3984ac00f None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:36.603095 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:DELETE=3,keystone:SELECT=96,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:36.634295 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd54a233-6532-409e-9eed-f9e396a3cb62 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:36.636539 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52b8198-87ad-43e3-92fd-7bae3e01fc73 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:36.637147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52b8198-87ad-43e3-92fd-7bae3e01fc73 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:36.637427 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52b8198-87ad-43e3-92fd-7bae3e01fc73 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:36.637692 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52b8198-87ad-43e3-92fd-7bae3e01fc73 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:36.638885 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f98ace-21b8-46fb-8140-095c1e9e770e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:36.639481 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f98ace-21b8-46fb-8140-095c1e9e770e admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:36.639764 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f98ace-21b8-46fb-8140-095c1e9e770e admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:36.640087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f98ace-21b8-46fb-8140-095c1e9e770e admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:36.640717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f98ace-21b8-46fb-8140-095c1e9e770e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:36.642643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f98ace-21b8-46fb-8140-095c1e9e770e admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:36.715225 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52b8198-87ad-43e3-92fd-7bae3e01fc73 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:36.716746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52b8198-87ad-43e3-92fd-7bae3e01fc73 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:36.734117 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 912/1805] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:36 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:36.742687 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 894/1806] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:36 2022] POST /identity/v3/projects => generated 365 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:02:36.743901 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52b8198-87ad-43e3-92fd-7bae3e01fc73 service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:36.744151 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:36.754586 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52b8198-87ad-43e3-92fd-7bae3e01fc73 service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:36.813473 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc9e67a-00b2-4da0-9809-7e4db413e53d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:36.814141 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc9e67a-00b2-4da0-9809-7e4db413e53d admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:36.814394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc9e67a-00b2-4da0-9809-7e4db413e53d admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:36.814668 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc9e67a-00b2-4da0-9809-7e4db413e53d admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:36.815303 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc9e67a-00b2-4da0-9809-7e4db413e53d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:36.817304 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc9e67a-00b2-4da0-9809-7e4db413e53d admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:36.934416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 913/1807] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:36 2022] POST /identity/v3/users => generated 289 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:36.938448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f98ace-21b8-46fb-8140-095c1e9e770e admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:36.948558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f98ace-21b8-46fb-8140-095c1e9e770e admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:36.953537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c51c2b1-b011-41e6-a706-ea83239f009e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:36.955013 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c51c2b1-b011-41e6-a706-ea83239f009e admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:36.955013 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c51c2b1-b011-41e6-a706-ea83239f009e admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:36.955013 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c51c2b1-b011-41e6-a706-ea83239f009e admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:36.955426 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c51c2b1-b011-41e6-a706-ea83239f009e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:36.957148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c51c2b1-b011-41e6-a706-ea83239f009e admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:36.968008 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 895/1808] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:36.972070 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc9e67a-00b2-4da0-9809-7e4db413e53d admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:36.983218 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc9e67a-00b2-4da0-9809-7e4db413e53d admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:36.988494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32e6841-b541-4329-961e-720ae343dc11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:36.990107 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32e6841-b541-4329-961e-720ae343dc11 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:36.990107 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32e6841-b541-4329-961e-720ae343dc11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:36.990107 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32e6841-b541-4329-961e-720ae343dc11 admin admin] PATH_INFO: `/v3/projects/75a7e741fe094d57a6c751714e82f6e9/users/97e850b435b64bb1ad1837e8ff65e750/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:37.005955 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32e6841-b541-4329-961e-720ae343dc11 admin admin] RBAC: Authorizing `identity:create_grant(project_id=75a7e741fe094d57a6c751714e82f6e9, user_id=97e850b435b64bb1ad1837e8ff65e750, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:37.008597 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32e6841-b541-4329-961e-720ae343dc11 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:37.050221 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c51c2b1-b011-41e6-a706-ea83239f009e admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:37.112450 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c51c2b1-b011-41e6-a706-ea83239f009e admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:37.119482 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 914/1809] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:36 2022] PUT /identity/v3/projects/75a7e741fe094d57a6c751714e82f6e9/users/97e850b435b64bb1ad1837e8ff65e750/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:37.124932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d25036-cf2c-4a6a-8fed-756f3430e1e7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:37.124932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d25036-cf2c-4a6a-8fed-756f3430e1e7 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:37.124932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d25036-cf2c-4a6a-8fed-756f3430e1e7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:37.124932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d25036-cf2c-4a6a-8fed-756f3430e1e7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:37.125954 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32e6841-b541-4329-961e-720ae343dc11 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:37.212246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d25036-cf2c-4a6a-8fed-756f3430e1e7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:37.214307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d25036-cf2c-4a6a-8fed-756f3430e1e7 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:37.232418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32e6841-b541-4329-961e-720ae343dc11 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:37.232535 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 896/1810] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:37 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:37.235930 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edcf3c0-e6e5-4a08-8c26-13140a8a98fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:37.236661 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edcf3c0-e6e5-4a08-8c26-13140a8a98fc admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:37.237062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edcf3c0-e6e5-4a08-8c26-13140a8a98fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:37.237521 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edcf3c0-e6e5-4a08-8c26-13140a8a98fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:37.238278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edcf3c0-e6e5-4a08-8c26-13140a8a98fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:37.242786 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edcf3c0-e6e5-4a08-8c26-13140a8a98fc admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:37.257991 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 915/1811] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:37 2022] GET /identity/v3/roles => generated 1420 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:02:37.268721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d25036-cf2c-4a6a-8fed-756f3430e1e7 service cinder] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:37.291253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d25036-cf2c-4a6a-8fed-756f3430e1e7 service cinder] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:37.291253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b50d4e-c33e-47b5-8868-bf291f195c3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:37.291253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b50d4e-c33e-47b5-8868-bf291f195c3b admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:37.291253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b50d4e-c33e-47b5-8868-bf291f195c3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:37.291663 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b50d4e-c33e-47b5-8868-bf291f195c3b admin admin] PATH_INFO: `/v3/projects/75a7e741fe094d57a6c751714e82f6e9/users/97e850b435b64bb1ad1837e8ff65e750/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:37.294038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b50d4e-c33e-47b5-8868-bf291f195c3b admin admin] RBAC: Authorizing `identity:create_grant(project_id=75a7e741fe094d57a6c751714e82f6e9, user_id=97e850b435b64bb1ad1837e8ff65e750, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:37.297810 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b50d4e-c33e-47b5-8868-bf291f195c3b admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:37.311985 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 897/1812] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:37 2022] PUT /identity/v3/projects/75a7e741fe094d57a6c751714e82f6e9/users/97e850b435b64bb1ad1837e8ff65e750/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:37.323532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f5ee7e-2500-4bfd-a7a4-5d5eaedba30f None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:37.323822 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f5ee7e-2500-4bfd-a7a4-5d5eaedba30f None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:37.324100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f5ee7e-2500-4bfd-a7a4-5d5eaedba30f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:37.353839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-40f5ee7e-2500-4bfd-a7a4-5d5eaedba30f None None] MFA Rules not processed for user `97e850b435b64bb1ad1837e8ff65e750`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:37.355059 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-40f5ee7e-2500-4bfd-a7a4-5d5eaedba30f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:37.391195 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b50d4e-c33e-47b5-8868-bf291f195c3b admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:37.433471 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 916/1813] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:37 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:37.445252 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f5ee7e-2500-4bfd-a7a4-5d5eaedba30f None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:37.446676 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-40f5ee7e-2500-4bfd-a7a4-5d5eaedba30f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:37.470185 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b50d4e-c33e-47b5-8868-bf291f195c3b admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:37.475087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5204d5-4092-4e6a-b112-34d9ec618735 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:37.475720 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5204d5-4092-4e6a-b112-34d9ec618735 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:37.476029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5204d5-4092-4e6a-b112-34d9ec618735 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:37.476347 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5204d5-4092-4e6a-b112-34d9ec618735 admin admin] PATH_INFO: `/v3/users/2f09d40387cd4e1d844df434a197da99` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:37.477446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5204d5-4092-4e6a-b112-34d9ec618735 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2f09d40387cd4e1d844df434a197da99)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:37.479415 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5204d5-4092-4e6a-b112-34d9ec618735 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:37.507050 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f5ee7e-2500-4bfd-a7a4-5d5eaedba30f None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:37.511076 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a300395-2dbf-438e-b482-3d574e1fff53 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:37.511791 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a300395-2dbf-438e-b482-3d574e1fff53 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:37.512144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a300395-2dbf-438e-b482-3d574e1fff53 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:37.512573 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a300395-2dbf-438e-b482-3d574e1fff53 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:37.524551 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a300395-2dbf-438e-b482-3d574e1fff53 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:37.526169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a300395-2dbf-438e-b482-3d574e1fff53 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:37.537842 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 917/1814] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:37 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:02:37.672357 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-6b5204d5-4092-4e6a-b112-34d9ec618735 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2f09d40387cd4e1d844df434a197da99'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:37.722167 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-6b5204d5-4092-4e6a-b112-34d9ec618735 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2f09d40387cd4e1d844df434a197da99'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:37.722904 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-6b5204d5-4092-4e6a-b112-34d9ec618735 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2f09d40387cd4e1d844df434a197da99'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:37.740938 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-6b5204d5-4092-4e6a-b112-34d9ec618735 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2f09d40387cd4e1d844df434a197da99'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:37.748443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-6b5204d5-4092-4e6a-b112-34d9ec618735 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2f09d40387cd4e1d844df434a197da99'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:37.768651 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 898/1815] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:37 2022] DELETE /identity/v3/users/2f09d40387cd4e1d844df434a197da99 => generated 0 bytes in 378 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:37.773434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b787fdd-5b3b-4b83-9024-57a41d41aa69 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:37.773720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b787fdd-5b3b-4b83-9024-57a41d41aa69 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:37.773996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b787fdd-5b3b-4b83-9024-57a41d41aa69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:37.778081 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5204d5-4092-4e6a-b112-34d9ec618735 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:37.785652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-6b5204d5-4092-4e6a-b112-34d9ec618735 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:37.797060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-0b787fdd-5b3b-4b83-9024-57a41d41aa69 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:37.807466 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-0b787fdd-5b3b-4b83-9024-57a41d41aa69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:37.874509 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5204d5-4092-4e6a-b112-34d9ec618735 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:37.879454 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49da228-3e2a-4ba1-9c7d-fbbb7a9716d9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:37.880829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49da228-3e2a-4ba1-9c7d-fbbb7a9716d9 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:37.881122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49da228-3e2a-4ba1-9c7d-fbbb7a9716d9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:37.881622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49da228-3e2a-4ba1-9c7d-fbbb7a9716d9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:37.883487 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-b49da228-3e2a-4ba1-9c7d-fbbb7a9716d9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:37.894470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 918/1816] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:37.910639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b787fdd-5b3b-4b83-9024-57a41d41aa69 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:37.910639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-0b787fdd-5b3b-4b83-9024-57a41d41aa69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:38.001899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49da228-3e2a-4ba1-9c7d-fbbb7a9716d9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:38.012455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49da228-3e2a-4ba1-9c7d-fbbb7a9716d9 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:38.015199 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b787fdd-5b3b-4b83-9024-57a41d41aa69 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:38.019559 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bb207c-237c-4af3-bba5-08657ea0322a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:38.021473 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bb207c-237c-4af3-bba5-08657ea0322a service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:38.022052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bb207c-237c-4af3-bba5-08657ea0322a service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:38.022450 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91bb207c-237c-4af3-bba5-08657ea0322a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:38.036826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bb207c-237c-4af3-bba5-08657ea0322a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:38.039985 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91bb207c-237c-4af3-bba5-08657ea0322a service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:38.047679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 899/1817] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:37 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 269 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:38.053091 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 919/1818] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:38.123319 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:38.316441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:DELETE=1,keystone:SELECT=98,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:38.538314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49da228-3e2a-4ba1-9c7d-fbbb7a9716d9 service cinder] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:38.540794 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-b49da228-3e2a-4ba1-9c7d-fbbb7a9716d9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:38.588247 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49da228-3e2a-4ba1-9c7d-fbbb7a9716d9 service cinder] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:38.602654 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38146ef1-296f-44ef-bcff-4f9dfe71b5b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:38.604230 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38146ef1-296f-44ef-bcff-4f9dfe71b5b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:38.604514 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38146ef1-296f-44ef-bcff-4f9dfe71b5b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:38.604780 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38146ef1-296f-44ef-bcff-4f9dfe71b5b9 admin admin] PATH_INFO: `/v3/users/83e5475d545c418cb1b1cb0a18447510` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:38.607748 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38146ef1-296f-44ef-bcff-4f9dfe71b5b9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=83e5475d545c418cb1b1cb0a18447510)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:38.610146 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38146ef1-296f-44ef-bcff-4f9dfe71b5b9 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:38.681479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:38.722439 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bb207c-237c-4af3-bba5-08657ea0322a service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:38.725664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-91bb207c-237c-4af3-bba5-08657ea0322a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:38.748850 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91bb207c-237c-4af3-bba5-08657ea0322a service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:38.777463 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd0e25d-98c8-4ad2-90d6-2d2503fb2ce1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:38.778362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd0e25d-98c8-4ad2-90d6-2d2503fb2ce1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:38.778827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd0e25d-98c8-4ad2-90d6-2d2503fb2ce1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:38.779253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd0e25d-98c8-4ad2-90d6-2d2503fb2ce1 admin admin] PATH_INFO: `/v3/projects/82a6b6347cee452782d71bac772d0e3f` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:38.781192 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd0e25d-98c8-4ad2-90d6-2d2503fb2ce1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=82a6b6347cee452782d71bac772d0e3f)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:38.785023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecd0e25d-98c8-4ad2-90d6-2d2503fb2ce1 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:38.853554 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-38146ef1-296f-44ef-bcff-4f9dfe71b5b9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '83e5475d545c418cb1b1cb0a18447510'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:38.977745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-38146ef1-296f-44ef-bcff-4f9dfe71b5b9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '83e5475d545c418cb1b1cb0a18447510'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:38.978629 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-38146ef1-296f-44ef-bcff-4f9dfe71b5b9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '83e5475d545c418cb1b1cb0a18447510'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:39.001828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ecd0e25d-98c8-4ad2-90d6-2d2503fb2ce1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '82a6b6347cee452782d71bac772d0e3f', 'request_id': 'req-ecd0e25d-98c8-4ad2-90d6-2d2503fb2ce1'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:39.019288 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-38146ef1-296f-44ef-bcff-4f9dfe71b5b9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '83e5475d545c418cb1b1cb0a18447510'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:39.044589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-38146ef1-296f-44ef-bcff-4f9dfe71b5b9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '83e5475d545c418cb1b1cb0a18447510'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:39.053146 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 900/1819] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:38 2022] DELETE /identity/v3/users/83e5475d545c418cb1b1cb0a18447510 => generated 0 bytes in 516 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:39.060804 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4abb90-5c91-4d10-8ba6-adde885b7676 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:39.061164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4abb90-5c91-4d10-8ba6-adde885b7676 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:39.061443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4abb90-5c91-4d10-8ba6-adde885b7676 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:39.084890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-dc4abb90-5c91-4d10-8ba6-adde885b7676 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:39.086421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-dc4abb90-5c91-4d10-8ba6-adde885b7676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:39.107746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ecd0e25d-98c8-4ad2-90d6-2d2503fb2ce1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '82a6b6347cee452782d71bac772d0e3f', 'request_id': 'req-ecd0e25d-98c8-4ad2-90d6-2d2503fb2ce1'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:39.115587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ecd0e25d-98c8-4ad2-90d6-2d2503fb2ce1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '82a6b6347cee452782d71bac772d0e3f', 'request_id': 'req-ecd0e25d-98c8-4ad2-90d6-2d2503fb2ce1'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:39.133555 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ecd0e25d-98c8-4ad2-90d6-2d2503fb2ce1 admin admin] The token cache is being invalidate because project 82a6b6347cee452782d71bac772d0e3f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:39.133829 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ecd0e25d-98c8-4ad2-90d6-2d2503fb2ce1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:39.134701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ecd0e25d-98c8-4ad2-90d6-2d2503fb2ce1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:39.136567 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 920/1820] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:38 2022] DELETE /identity/v3/projects/82a6b6347cee452782d71bac772d0e3f => generated 0 bytes in 417 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:02:39.141233 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:39.141350 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd0e25d-98c8-4ad2-90d6-2d2503fb2ce1 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:39.142931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-ecd0e25d-98c8-4ad2-90d6-2d2503fb2ce1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:39.160404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 901/1821] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:02:39.175295 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4abb90-5c91-4d10-8ba6-adde885b7676 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:39.188134 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-dc4abb90-5c91-4d10-8ba6-adde885b7676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:39.285616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd0e25d-98c8-4ad2-90d6-2d2503fb2ce1 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:39.296145 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccb47ec-7a3a-4f8b-a8a4-cdc4ff99747f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:39.296911 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccb47ec-7a3a-4f8b-a8a4-cdc4ff99747f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:39.297197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccb47ec-7a3a-4f8b-a8a4-cdc4ff99747f admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:39.297770 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccb47ec-7a3a-4f8b-a8a4-cdc4ff99747f admin admin] PATH_INFO: `/v3/users/1a5c2019a1cb487f84d706365f30ec8f` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:39.305771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccb47ec-7a3a-4f8b-a8a4-cdc4ff99747f admin admin] RBAC: Authorizing `identity:delete_user(user_id=1a5c2019a1cb487f84d706365f30ec8f)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:39.307786 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccb47ec-7a3a-4f8b-a8a4-cdc4ff99747f admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:39.351211 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4abb90-5c91-4d10-8ba6-adde885b7676 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:39.365449 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289b0cb0-629f-41ab-80ba-c149b047cba5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:39.366069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289b0cb0-629f-41ab-80ba-c149b047cba5 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:39.366371 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289b0cb0-629f-41ab-80ba-c149b047cba5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:39.366886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289b0cb0-629f-41ab-80ba-c149b047cba5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:39.379939 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289b0cb0-629f-41ab-80ba-c149b047cba5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:39.380229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:39.382837 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289b0cb0-629f-41ab-80ba-c149b047cba5 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:39.403392 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 902/1822] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:39.456322 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:39.571548 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9ccb47ec-7a3a-4f8b-a8a4-cdc4ff99747f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1a5c2019a1cb487f84d706365f30ec8f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:39.642485 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9ccb47ec-7a3a-4f8b-a8a4-cdc4ff99747f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1a5c2019a1cb487f84d706365f30ec8f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:39.666009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9ccb47ec-7a3a-4f8b-a8a4-cdc4ff99747f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1a5c2019a1cb487f84d706365f30ec8f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:39.667369 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9ccb47ec-7a3a-4f8b-a8a4-cdc4ff99747f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1a5c2019a1cb487f84d706365f30ec8f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:39.686559 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9ccb47ec-7a3a-4f8b-a8a4-cdc4ff99747f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1a5c2019a1cb487f84d706365f30ec8f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:39.713246 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 921/1823] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:39 2022] DELETE /identity/v3/users/1a5c2019a1cb487f84d706365f30ec8f => generated 0 bytes in 570 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:02:39.717123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:39.745628 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:40.004764 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289b0cb0-629f-41ab-80ba-c149b047cba5 service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:40.006777 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-289b0cb0-629f-41ab-80ba-c149b047cba5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:40.033337 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289b0cb0-629f-41ab-80ba-c149b047cba5 service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:40.108710 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88cec4e2-f061-46ae-b9c2-3d0a61d72abc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:40.109371 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cec4e2-f061-46ae-b9c2-3d0a61d72abc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:40.109648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cec4e2-f061-46ae-b9c2-3d0a61d72abc admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:40.109917 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cec4e2-f061-46ae-b9c2-3d0a61d72abc admin admin] PATH_INFO: `/v3/projects/59a3cb3df9fe443f90dfd489e581278c` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:40.111159 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88cec4e2-f061-46ae-b9c2-3d0a61d72abc admin admin] RBAC: Authorizing `identity:delete_project(project_id=59a3cb3df9fe443f90dfd489e581278c)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:40.113488 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88cec4e2-f061-46ae-b9c2-3d0a61d72abc admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:40.225517 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:40.241279 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-88cec4e2-f061-46ae-b9c2-3d0a61d72abc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '59a3cb3df9fe443f90dfd489e581278c', 'request_id': 'req-88cec4e2-f061-46ae-b9c2-3d0a61d72abc'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:40.333490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-88cec4e2-f061-46ae-b9c2-3d0a61d72abc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '59a3cb3df9fe443f90dfd489e581278c', 'request_id': 'req-88cec4e2-f061-46ae-b9c2-3d0a61d72abc'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:40.347702 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-88cec4e2-f061-46ae-b9c2-3d0a61d72abc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '59a3cb3df9fe443f90dfd489e581278c', 'request_id': 'req-88cec4e2-f061-46ae-b9c2-3d0a61d72abc'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:40.362642 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-88cec4e2-f061-46ae-b9c2-3d0a61d72abc admin admin] The token cache is being invalidate because project 59a3cb3df9fe443f90dfd489e581278c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:40.363373 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-88cec4e2-f061-46ae-b9c2-3d0a61d72abc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:40.374842 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-88cec4e2-f061-46ae-b9c2-3d0a61d72abc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:40.377526 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 903/1824] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:40 2022] DELETE /identity/v3/projects/59a3cb3df9fe443f90dfd489e581278c => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:40.381908 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccb47ec-7a3a-4f8b-a8a4-cdc4ff99747f admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:40.383465 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-9ccb47ec-7a3a-4f8b-a8a4-cdc4ff99747f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:40.399605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccb47ec-7a3a-4f8b-a8a4-cdc4ff99747f admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:40.454658 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:40.497902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed28822-9402-4fbd-90c6-0fd1a206e634 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:40.498720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed28822-9402-4fbd-90c6-0fd1a206e634 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:40.499068 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed28822-9402-4fbd-90c6-0fd1a206e634 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:40.499503 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed28822-9402-4fbd-90c6-0fd1a206e634 admin admin] PATH_INFO: `/v3/users/ac803da0c82141a78b9093c1e360dafe` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:40.500942 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed28822-9402-4fbd-90c6-0fd1a206e634 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ac803da0c82141a78b9093c1e360dafe)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:40.502524 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88cec4e2-f061-46ae-b9c2-3d0a61d72abc admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:40.503579 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed28822-9402-4fbd-90c6-0fd1a206e634 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:40.504788 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-88cec4e2-f061-46ae-b9c2-3d0a61d72abc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:40.583376 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88cec4e2-f061-46ae-b9c2-3d0a61d72abc admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:40.595980 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fb4d482-702e-4200-9164-893cda7460ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:40.595980 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb4d482-702e-4200-9164-893cda7460ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:40.595980 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb4d482-702e-4200-9164-893cda7460ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:40.595980 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb4d482-702e-4200-9164-893cda7460ae admin admin] PATH_INFO: `/v3/projects/fc72405ff8bc4e3bb977083d341f1f5c` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:40.596459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb4d482-702e-4200-9164-893cda7460ae admin admin] RBAC: Authorizing `identity:delete_project(project_id=fc72405ff8bc4e3bb977083d341f1f5c)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:40.599022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fb4d482-702e-4200-9164-893cda7460ae admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:40.797102 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-aed28822-9402-4fbd-90c6-0fd1a206e634 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ac803da0c82141a78b9093c1e360dafe'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:40.798156 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-4fb4d482-702e-4200-9164-893cda7460ae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fc72405ff8bc4e3bb977083d341f1f5c', 'request_id': 'req-4fb4d482-702e-4200-9164-893cda7460ae'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:40.865638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-aed28822-9402-4fbd-90c6-0fd1a206e634 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ac803da0c82141a78b9093c1e360dafe'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:40.888529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-aed28822-9402-4fbd-90c6-0fd1a206e634 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ac803da0c82141a78b9093c1e360dafe'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:40.902627 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-aed28822-9402-4fbd-90c6-0fd1a206e634 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ac803da0c82141a78b9093c1e360dafe'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:40.927422 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-aed28822-9402-4fbd-90c6-0fd1a206e634 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ac803da0c82141a78b9093c1e360dafe'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:40.961430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 922/1825] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:40 2022] DELETE /identity/v3/users/ac803da0c82141a78b9093c1e360dafe => generated 0 bytes in 576 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:40.997294 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-4fb4d482-702e-4200-9164-893cda7460ae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fc72405ff8bc4e3bb977083d341f1f5c', 'request_id': 'req-4fb4d482-702e-4200-9164-893cda7460ae'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:41.006681 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-4fb4d482-702e-4200-9164-893cda7460ae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fc72405ff8bc4e3bb977083d341f1f5c', 'request_id': 'req-4fb4d482-702e-4200-9164-893cda7460ae'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:41.048670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-4fb4d482-702e-4200-9164-893cda7460ae admin admin] The token cache is being invalidate because project fc72405ff8bc4e3bb977083d341f1f5c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:41.052122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-4fb4d482-702e-4200-9164-893cda7460ae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:41.054561 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-4fb4d482-702e-4200-9164-893cda7460ae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:41.059057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 904/1826] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:40 2022] DELETE /identity/v3/projects/fc72405ff8bc4e3bb977083d341f1f5c => generated 0 bytes in 557 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:02:41.064861 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc612bff-dc90-48b2-bc9a-ae6f2e723876 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:41.065470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc612bff-dc90-48b2-bc9a-ae6f2e723876 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:41.065914 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc612bff-dc90-48b2-bc9a-ae6f2e723876 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:41.103778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-fc612bff-dc90-48b2-bc9a-ae6f2e723876 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:41.106147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-fc612bff-dc90-48b2-bc9a-ae6f2e723876 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:41.206907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 923/1827] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:41.228331 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401f7676-aaa6-4a32-af30-677bfab2181d None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:41.228618 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401f7676-aaa6-4a32-af30-677bfab2181d None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:41.228891 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401f7676-aaa6-4a32-af30-677bfab2181d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:41.256817 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-401f7676-aaa6-4a32-af30-677bfab2181d None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:41.258807 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-401f7676-aaa6-4a32-af30-677bfab2181d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:41.290544 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 905/1828] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:41.291876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc612bff-dc90-48b2-bc9a-ae6f2e723876 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:41.306683 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc612bff-dc90-48b2-bc9a-ae6f2e723876 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:41.311887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd1231f-4a98-4711-9081-8b1c75c4bc04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:41.313599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd1231f-4a98-4711-9081-8b1c75c4bc04 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:41.313599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd1231f-4a98-4711-9081-8b1c75c4bc04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:41.313599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd1231f-4a98-4711-9081-8b1c75c4bc04 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:41.314423 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd1231f-4a98-4711-9081-8b1c75c4bc04 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:41.315202 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd1231f-4a98-4711-9081-8b1c75c4bc04 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:41.317868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcd1231f-4a98-4711-9081-8b1c75c4bc04 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:41.334369 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 924/1829] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:02:41.338086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401f7676-aaa6-4a32-af30-677bfab2181d None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:41.350763 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401f7676-aaa6-4a32-af30-677bfab2181d None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:41.360021 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c02ee9-3288-46e7-9210-68f0e3e37812 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:41.360950 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c02ee9-3288-46e7-9210-68f0e3e37812 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:41.361389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c02ee9-3288-46e7-9210-68f0e3e37812 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:41.361800 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c02ee9-3288-46e7-9210-68f0e3e37812 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:41.362652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c02ee9-3288-46e7-9210-68f0e3e37812 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:41.365405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c02ee9-3288-46e7-9210-68f0e3e37812 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:41.442957 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:41.501622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 906/1830] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:41 2022] POST /identity/v3/projects => generated 385 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:41.503285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd1231f-4a98-4711-9081-8b1c75c4bc04 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:41.534013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd1231f-4a98-4711-9081-8b1c75c4bc04 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:41.593812 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:41.686028 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b49f40-2ba2-410f-884c-4ab0ba59405d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:41.686912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b49f40-2ba2-410f-884c-4ab0ba59405d admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:41.687196 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b49f40-2ba2-410f-884c-4ab0ba59405d admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:41.687488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5b49f40-2ba2-410f-884c-4ab0ba59405d admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:41.688143 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b49f40-2ba2-410f-884c-4ab0ba59405d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:41.690668 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5b49f40-2ba2-410f-884c-4ab0ba59405d admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:41.819948 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 925/1831] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:41 2022] POST /identity/v3/users => generated 293 bytes in 314 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:41.821532 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c02ee9-3288-46e7-9210-68f0e3e37812 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:41.851051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c02ee9-3288-46e7-9210-68f0e3e37812 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:41.869496 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8165460f-0228-4413-b88a-4f7cc038c7e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:41.872063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8165460f-0228-4413-b88a-4f7cc038c7e0 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:41.872063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8165460f-0228-4413-b88a-4f7cc038c7e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:41.872063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8165460f-0228-4413-b88a-4f7cc038c7e0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:41.872063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8165460f-0228-4413-b88a-4f7cc038c7e0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:41.872976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8165460f-0228-4413-b88a-4f7cc038c7e0 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:41.928861 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 907/1832] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:41 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:41.930594 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b49f40-2ba2-410f-884c-4ab0ba59405d admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:41.948621 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5b49f40-2ba2-410f-884c-4ab0ba59405d admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:41.973419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5a1990-3c3b-4ff1-bf42-46395688fd68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:41.974357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5a1990-3c3b-4ff1-bf42-46395688fd68 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:41.974844 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5a1990-3c3b-4ff1-bf42-46395688fd68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:41.975336 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5a1990-3c3b-4ff1-bf42-46395688fd68 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:41.976169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a5a1990-3c3b-4ff1-bf42-46395688fd68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:41.979975 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a5a1990-3c3b-4ff1-bf42-46395688fd68 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:42.021740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8165460f-0228-4413-b88a-4f7cc038c7e0 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:42.028267 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 926/1833] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:41 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:42.032610 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8165460f-0228-4413-b88a-4f7cc038c7e0 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:42.040975 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf70198-4758-49d0-9501-d9cf5d9c16b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:42.040975 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf70198-4758-49d0-9501-d9cf5d9c16b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:42.057421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf70198-4758-49d0-9501-d9cf5d9c16b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:42.058753 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf70198-4758-49d0-9501-d9cf5d9c16b0 admin admin] PATH_INFO: `/v3/projects/a33bc4bba7f5413285659421477c1081/users/fb90ed7977fc4a7dabb4fbaff61fe95e/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:42.083493 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf70198-4758-49d0-9501-d9cf5d9c16b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a33bc4bba7f5413285659421477c1081, user_id=fb90ed7977fc4a7dabb4fbaff61fe95e, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:42.087174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf70198-4758-49d0-9501-d9cf5d9c16b0 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:42.120939 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5a1990-3c3b-4ff1-bf42-46395688fd68 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:42.121633 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-1a5a1990-3c3b-4ff1-bf42-46395688fd68 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:42.147573 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5a1990-3c3b-4ff1-bf42-46395688fd68 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:42.153654 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba62035-28fe-488c-9e45-12e44d006a07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:42.154933 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba62035-28fe-488c-9e45-12e44d006a07 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:42.154933 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba62035-28fe-488c-9e45-12e44d006a07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:42.154933 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba62035-28fe-488c-9e45-12e44d006a07 admin admin] PATH_INFO: `/v3/projects/59c7d00de35544e1ad06e8b4f290e6c0` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:42.156392 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba62035-28fe-488c-9e45-12e44d006a07 admin admin] RBAC: Authorizing `identity:delete_project(project_id=59c7d00de35544e1ad06e8b4f290e6c0)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:42.158376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba62035-28fe-488c-9e45-12e44d006a07 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:42.209751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 908/1834] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:42 2022] PUT /identity/v3/projects/a33bc4bba7f5413285659421477c1081/users/fb90ed7977fc4a7dabb4fbaff61fe95e/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:42.211396 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9908b7-91fe-46f3-9f5e-7804779a3896 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:42.212394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9908b7-91fe-46f3-9f5e-7804779a3896 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:42.213233 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a9908b7-91fe-46f3-9f5e-7804779a3896 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:42.244821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-2a9908b7-91fe-46f3-9f5e-7804779a3896 None None] MFA Rules not processed for user `fb90ed7977fc4a7dabb4fbaff61fe95e`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:42.246086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2a9908b7-91fe-46f3-9f5e-7804779a3896 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:42.313356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7ba62035-28fe-488c-9e45-12e44d006a07 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '59c7d00de35544e1ad06e8b4f290e6c0', 'request_id': 'req-7ba62035-28fe-488c-9e45-12e44d006a07'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:42.321680 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 909/1835] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:42 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:42.325726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9908b7-91fe-46f3-9f5e-7804779a3896 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:42.379035 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7ba62035-28fe-488c-9e45-12e44d006a07 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '59c7d00de35544e1ad06e8b4f290e6c0', 'request_id': 'req-7ba62035-28fe-488c-9e45-12e44d006a07'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:42.387157 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7ba62035-28fe-488c-9e45-12e44d006a07 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '59c7d00de35544e1ad06e8b4f290e6c0', 'request_id': 'req-7ba62035-28fe-488c-9e45-12e44d006a07'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:42.397858 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a9908b7-91fe-46f3-9f5e-7804779a3896 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:42.399876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7ba62035-28fe-488c-9e45-12e44d006a07 admin admin] The token cache is being invalidate because project 59c7d00de35544e1ad06e8b4f290e6c0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:42.400245 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7ba62035-28fe-488c-9e45-12e44d006a07 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:42.400957 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7ba62035-28fe-488c-9e45-12e44d006a07 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:42.403093 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 927/1836] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:42 2022] DELETE /identity/v3/projects/59c7d00de35544e1ad06e8b4f290e6c0 => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:02:42.408175 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bc1aeb-fc2f-4291-9a85-9450a50e75e0 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:42.409128 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bc1aeb-fc2f-4291-9a85-9450a50e75e0 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:42.409128 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bc1aeb-fc2f-4291-9a85-9450a50e75e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:42.437004 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-d6bc1aeb-fc2f-4291-9a85-9450a50e75e0 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:42.439119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d6bc1aeb-fc2f-4291-9a85-9450a50e75e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:42.508923 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a588539-dd26-43ec-a456-36145558085e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:42.510856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a588539-dd26-43ec-a456-36145558085e admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:42.511400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a588539-dd26-43ec-a456-36145558085e admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:42.511688 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a588539-dd26-43ec-a456-36145558085e admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:42.513248 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a588539-dd26-43ec-a456-36145558085e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:42.516345 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a588539-dd26-43ec-a456-36145558085e admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:42.546379 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 928/1837] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:02:42.547943 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85a136e-0ce0-40f7-9093-7e4e96f05420 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:42.548288 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85a136e-0ce0-40f7-9093-7e4e96f05420 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:42.554126 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85a136e-0ce0-40f7-9093-7e4e96f05420 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:42.569863 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-e85a136e-0ce0-40f7-9093-7e4e96f05420 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:42.571208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-e85a136e-0ce0-40f7-9093-7e4e96f05420 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:42.580412 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 929/1838] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:42.588825 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85a136e-0ce0-40f7-9093-7e4e96f05420 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:42.594600 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85a136e-0ce0-40f7-9093-7e4e96f05420 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:42.600616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b87fcf6-f9da-43b1-9d81-589adf65833d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:42.601314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b87fcf6-f9da-43b1-9d81-589adf65833d admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:42.601669 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b87fcf6-f9da-43b1-9d81-589adf65833d admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:42.602304 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b87fcf6-f9da-43b1-9d81-589adf65833d admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:42.602816 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b87fcf6-f9da-43b1-9d81-589adf65833d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:42.603235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b87fcf6-f9da-43b1-9d81-589adf65833d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:42.605038 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b87fcf6-f9da-43b1-9d81-589adf65833d admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:42.617585 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 910/1839] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:42 2022] POST /identity/v3/projects => generated 387 bytes in 292 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:02:42.621695 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a588539-dd26-43ec-a456-36145558085e admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:42.627839 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-7a588539-dd26-43ec-a456-36145558085e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:42.632239 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 930/1840] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 14:02:42.636481 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b87fcf6-f9da-43b1-9d81-589adf65833d admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:42.648404 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b87fcf6-f9da-43b1-9d81-589adf65833d admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:42.702664 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a588539-dd26-43ec-a456-36145558085e admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:42.711498 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68caea16-bb13-42c6-bec1-43df20a615ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:42.712438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68caea16-bb13-42c6-bec1-43df20a615ee admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:42.715624 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68caea16-bb13-42c6-bec1-43df20a615ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:42.715746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea73e888-a82b-49a3-9644-597c50df627e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:42.716245 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68caea16-bb13-42c6-bec1-43df20a615ee admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:42.716807 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea73e888-a82b-49a3-9644-597c50df627e admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:42.717196 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea73e888-a82b-49a3-9644-597c50df627e admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:42.717318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68caea16-bb13-42c6-bec1-43df20a615ee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:42.717689 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea73e888-a82b-49a3-9644-597c50df627e admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:42.718587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea73e888-a82b-49a3-9644-597c50df627e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:42.719850 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68caea16-bb13-42c6-bec1-43df20a615ee admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:42.726312 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea73e888-a82b-49a3-9644-597c50df627e admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:42.867514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 931/1841] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:42 2022] POST /identity/v3/projects => generated 369 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:42.868336 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 911/1842] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:42 2022] POST /identity/v3/users => generated 300 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:42.869687 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68caea16-bb13-42c6-bec1-43df20a615ee admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:42.880725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d459f8a1-77dd-4e9c-a045-95b3e122d5c7 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:42.882746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d459f8a1-77dd-4e9c-a045-95b3e122d5c7 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:42.882945 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68caea16-bb13-42c6-bec1-43df20a615ee admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:42.884023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d459f8a1-77dd-4e9c-a045-95b3e122d5c7 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:42.885866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 932/1843] 10.222.0.62 () {54 vars in 921 bytes} [Fri Sep 23 14:02:42 2022] GET /identity => generated 270 bytes in 18 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:02:42.886799 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d459f8a1-77dd-4e9c-a045-95b3e122d5c7 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:42.966714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031fbe54-3123-419a-8b95-b7d6bf9a4d94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:42.967713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031fbe54-3123-419a-8b95-b7d6bf9a4d94 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:42.968350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031fbe54-3123-419a-8b95-b7d6bf9a4d94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:42.968695 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031fbe54-3123-419a-8b95-b7d6bf9a4d94 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:42.969340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031fbe54-3123-419a-8b95-b7d6bf9a4d94 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:42.972251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031fbe54-3123-419a-8b95-b7d6bf9a4d94 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:42.997613 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d459f8a1-77dd-4e9c-a045-95b3e122d5c7 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:43.014065 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80347478-2265-4524-8502-ccb029a95a37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:43.014907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80347478-2265-4524-8502-ccb029a95a37 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:43.015444 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80347478-2265-4524-8502-ccb029a95a37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:43.015721 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80347478-2265-4524-8502-ccb029a95a37 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:43.016443 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80347478-2265-4524-8502-ccb029a95a37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:43.018635 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80347478-2265-4524-8502-ccb029a95a37 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:43.046258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 933/1844] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:42 2022] GET /identity/v3/roles => generated 1420 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:43.053396 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd685cd-4ed2-4263-b50a-4a330818b15d None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:43.053691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd685cd-4ed2-4263-b50a-4a330818b15d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:43.054229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd685cd-4ed2-4263-b50a-4a330818b15d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:43.080739 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-ecd685cd-4ed2-4263-b50a-4a330818b15d None None] MFA Rules not processed for user `f01ed94dbb384a178ad0a45a6f3d8c17`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:43.081854 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-ecd685cd-4ed2-4263-b50a-4a330818b15d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:43.108097 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 912/1845] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:42 2022] POST /identity/v3/users => generated 285 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:43.110016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031fbe54-3123-419a-8b95-b7d6bf9a4d94 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:43.121584 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031fbe54-3123-419a-8b95-b7d6bf9a4d94 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:43.127744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0fa969-70d4-42bb-bc32-64ca2ad538af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:43.128418 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0fa969-70d4-42bb-bc32-64ca2ad538af admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:43.128697 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0fa969-70d4-42bb-bc32-64ca2ad538af admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:43.128968 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0fa969-70d4-42bb-bc32-64ca2ad538af admin admin] PATH_INFO: `/v3/projects/0e8fbbc2b5da4f769bfea42f5d94a625/users/bee1fcf80afb40ab9a68f16184908885/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:43.141998 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:43.152690 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0fa969-70d4-42bb-bc32-64ca2ad538af admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e8fbbc2b5da4f769bfea42f5d94a625, user_id=bee1fcf80afb40ab9a68f16184908885, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:43.155507 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0fa969-70d4-42bb-bc32-64ca2ad538af admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:43.173280 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 934/1846] 10.222.0.62 () {58 vars in 1018 bytes} [Fri Sep 23 14:02:43 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:43.174256 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd685cd-4ed2-4263-b50a-4a330818b15d None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:43.185836 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd685cd-4ed2-4263-b50a-4a330818b15d None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:43.191407 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ab4fab-47a0-401a-b29f-967d08867eaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:43.192121 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ab4fab-47a0-401a-b29f-967d08867eaf admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:43.192404 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ab4fab-47a0-401a-b29f-967d08867eaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:43.192672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ab4fab-47a0-401a-b29f-967d08867eaf admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:43.193358 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ab4fab-47a0-401a-b29f-967d08867eaf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:43.195032 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ab4fab-47a0-401a-b29f-967d08867eaf admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:43.224576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 935/1847] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:43 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:43.225515 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ab4fab-47a0-401a-b29f-967d08867eaf admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:43.227626 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f2ab4fab-47a0-401a-b29f-967d08867eaf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:43.259646 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ab4fab-47a0-401a-b29f-967d08867eaf admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:43.261767 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 913/1848] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:43 2022] PUT /identity/v3/projects/0e8fbbc2b5da4f769bfea42f5d94a625/users/bee1fcf80afb40ab9a68f16184908885/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:43.265666 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0fa969-70d4-42bb-bc32-64ca2ad538af admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:43.333198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36c2f9b-e83b-4fe3-8300-e644624c0df5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:43.333820 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36c2f9b-e83b-4fe3-8300-e644624c0df5 service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:43.334097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36c2f9b-e83b-4fe3-8300-e644624c0df5 service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:43.334387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36c2f9b-e83b-4fe3-8300-e644624c0df5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:43.345785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36c2f9b-e83b-4fe3-8300-e644624c0df5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:43.347540 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36c2f9b-e83b-4fe3-8300-e644624c0df5 service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:43.354171 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:43.360535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 936/1849] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:43 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:43.361722 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36c2f9b-e83b-4fe3-8300-e644624c0df5 service nova] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:43.396738 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0fa969-70d4-42bb-bc32-64ca2ad538af admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:43.401700 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4073eb7c-96ce-4f7f-89fb-ccabebc085e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:43.402478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4073eb7c-96ce-4f7f-89fb-ccabebc085e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:43.402761 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4073eb7c-96ce-4f7f-89fb-ccabebc085e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:43.403037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4073eb7c-96ce-4f7f-89fb-ccabebc085e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:43.403687 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4073eb7c-96ce-4f7f-89fb-ccabebc085e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:43.405290 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4073eb7c-96ce-4f7f-89fb-ccabebc085e8 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:43.417247 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 914/1850] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:43 2022] GET /identity/v3/roles => generated 1420 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:43.421429 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4073eb7c-96ce-4f7f-89fb-ccabebc085e8 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:43.432006 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4073eb7c-96ce-4f7f-89fb-ccabebc085e8 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:43.438052 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8891ee20-9494-4e50-9ba1-5cf6c778e059 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:43.438767 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8891ee20-9494-4e50-9ba1-5cf6c778e059 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:43.439360 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8891ee20-9494-4e50-9ba1-5cf6c778e059 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:43.439640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8891ee20-9494-4e50-9ba1-5cf6c778e059 admin admin] PATH_INFO: `/v3/projects/607dd350513840c7a6476644388b4a08/users/14c177b6fe5642d7aff1bd8bc049107a/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:43.455593 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8891ee20-9494-4e50-9ba1-5cf6c778e059 admin admin] RBAC: Authorizing `identity:create_grant(project_id=607dd350513840c7a6476644388b4a08, user_id=14c177b6fe5642d7aff1bd8bc049107a, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:43.458998 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8891ee20-9494-4e50-9ba1-5cf6c778e059 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:43.462183 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36c2f9b-e83b-4fe3-8300-e644624c0df5 service nova] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:43.467406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637afe48-03ae-47b6-88e3-ee1ed6f2df98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:43.468009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637afe48-03ae-47b6-88e3-ee1ed6f2df98 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:43.468289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637afe48-03ae-47b6-88e3-ee1ed6f2df98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:43.468582 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637afe48-03ae-47b6-88e3-ee1ed6f2df98 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:43.469194 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637afe48-03ae-47b6-88e3-ee1ed6f2df98 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:43.470805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637afe48-03ae-47b6-88e3-ee1ed6f2df98 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:43.481700 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 937/1851] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:43 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:43.486006 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637afe48-03ae-47b6-88e3-ee1ed6f2df98 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:43.496374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637afe48-03ae-47b6-88e3-ee1ed6f2df98 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:43.500997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fd5605-b8f3-46ff-bf48-c49a896748f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:43.501644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fd5605-b8f3-46ff-bf48-c49a896748f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:43.508720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fd5605-b8f3-46ff-bf48-c49a896748f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:43.508720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fd5605-b8f3-46ff-bf48-c49a896748f1 admin admin] PATH_INFO: `/v3/projects/0e8fbbc2b5da4f769bfea42f5d94a625/users/bee1fcf80afb40ab9a68f16184908885/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:43.508720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fd5605-b8f3-46ff-bf48-c49a896748f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e8fbbc2b5da4f769bfea42f5d94a625, user_id=bee1fcf80afb40ab9a68f16184908885, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:43.508720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fd5605-b8f3-46ff-bf48-c49a896748f1 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:43.518449 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 938/1852] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:43 2022] PUT /identity/v3/projects/0e8fbbc2b5da4f769bfea42f5d94a625/users/bee1fcf80afb40ab9a68f16184908885/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:02:43.529108 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462d256e-267c-4748-b8c8-534739da8842 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:43.529395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462d256e-267c-4748-b8c8-534739da8842 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:43.529668 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462d256e-267c-4748-b8c8-534739da8842 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:43.569373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-462d256e-267c-4748-b8c8-534739da8842 None None] MFA Rules not processed for user `bee1fcf80afb40ab9a68f16184908885`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:43.570809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-462d256e-267c-4748-b8c8-534739da8842 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:43.580024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 915/1853] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:43 2022] PUT /identity/v3/projects/607dd350513840c7a6476644388b4a08/users/14c177b6fe5642d7aff1bd8bc049107a/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:43.581620 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9aa90f-b86c-45c1-ad77-a53a7ae2599e None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:43.581893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9aa90f-b86c-45c1-ad77-a53a7ae2599e None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:43.582781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9aa90f-b86c-45c1-ad77-a53a7ae2599e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:43.642989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-cd9aa90f-b86c-45c1-ad77-a53a7ae2599e None None] MFA Rules not processed for user `14c177b6fe5642d7aff1bd8bc049107a`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:43.644283 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-cd9aa90f-b86c-45c1-ad77-a53a7ae2599e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:43.706411 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 939/1854] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:43 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:02:43.725146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462d256e-267c-4748-b8c8-534739da8842 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:43.726858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-462d256e-267c-4748-b8c8-534739da8842 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:43.734492 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 916/1855] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:43 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:43.737052 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9aa90f-b86c-45c1-ad77-a53a7ae2599e None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:43.776269 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:43.801183 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462d256e-267c-4748-b8c8-534739da8842 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:43.806428 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3b4522-35f4-4234-a9fd-6a554fbc23a4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:43.807055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3b4522-35f4-4234-a9fd-6a554fbc23a4 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:43.807333 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3b4522-35f4-4234-a9fd-6a554fbc23a4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:43.807606 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3b4522-35f4-4234-a9fd-6a554fbc23a4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:43.818904 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3b4522-35f4-4234-a9fd-6a554fbc23a4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:43.820390 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3b4522-35f4-4234-a9fd-6a554fbc23a4 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:43.831325 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 940/1856] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:43 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:02:43.842937 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9aa90f-b86c-45c1-ad77-a53a7ae2599e None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:43.842937 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d930162-2e84-4015-839f-13baa1e35dcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:43.842937 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d930162-2e84-4015-839f-13baa1e35dcd admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:43.842937 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d930162-2e84-4015-839f-13baa1e35dcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:43.843742 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d930162-2e84-4015-839f-13baa1e35dcd admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:43.843742 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d930162-2e84-4015-839f-13baa1e35dcd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:43.844065 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d930162-2e84-4015-839f-13baa1e35dcd admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:43.921641 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3b4522-35f4-4234-a9fd-6a554fbc23a4 service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:43.933892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3b4522-35f4-4234-a9fd-6a554fbc23a4 service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:43.945617 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 917/1857] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:43 2022] POST /identity/v3/projects => generated 369 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:43.945842 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae1df68-e291-44b9-812c-37a3bac0d26f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:43.945842 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae1df68-e291-44b9-812c-37a3bac0d26f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:43.947124 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d930162-2e84-4015-839f-13baa1e35dcd admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:43.947880 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae1df68-e291-44b9-812c-37a3bac0d26f admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:43.949589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae1df68-e291-44b9-812c-37a3bac0d26f admin admin] PATH_INFO: `/v3/users/fb90ed7977fc4a7dabb4fbaff61fe95e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:43.952128 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae1df68-e291-44b9-812c-37a3bac0d26f admin admin] RBAC: Authorizing `identity:delete_user(user_id=fb90ed7977fc4a7dabb4fbaff61fe95e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:43.955045 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae1df68-e291-44b9-812c-37a3bac0d26f admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:44.021994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d930162-2e84-4015-839f-13baa1e35dcd admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:44.030196 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d9890a-44ad-4435-acee-35c52410e74e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:44.030967 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d9890a-44ad-4435-acee-35c52410e74e admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:44.031250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d9890a-44ad-4435-acee-35c52410e74e admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:44.031518 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d9890a-44ad-4435-acee-35c52410e74e admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:44.032154 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d9890a-44ad-4435-acee-35c52410e74e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:44.034588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d9890a-44ad-4435-acee-35c52410e74e admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:44.152087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 918/1858] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:43 2022] POST /identity/v3/users => generated 291 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:44.163724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d9890a-44ad-4435-acee-35c52410e74e admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:44.183658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d9890a-44ad-4435-acee-35c52410e74e admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:44.185723 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-eae1df68-e291-44b9-812c-37a3bac0d26f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fb90ed7977fc4a7dabb4fbaff61fe95e'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:44.196212 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402c38e5-ce49-4510-a2b6-8b2744fb0977 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:44.196903 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402c38e5-ce49-4510-a2b6-8b2744fb0977 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:44.197195 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402c38e5-ce49-4510-a2b6-8b2744fb0977 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:44.197460 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402c38e5-ce49-4510-a2b6-8b2744fb0977 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:44.198362 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402c38e5-ce49-4510-a2b6-8b2744fb0977 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:44.204350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402c38e5-ce49-4510-a2b6-8b2744fb0977 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:44.224408 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 919/1859] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:44 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:02:44.233743 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402c38e5-ce49-4510-a2b6-8b2744fb0977 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:44.243944 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:44.248328 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402c38e5-ce49-4510-a2b6-8b2744fb0977 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:44.260154 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605fb49a-4918-46ab-ad73-1f42c72e7374 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:44.260846 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605fb49a-4918-46ab-ad73-1f42c72e7374 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:44.261123 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605fb49a-4918-46ab-ad73-1f42c72e7374 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:44.266770 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605fb49a-4918-46ab-ad73-1f42c72e7374 admin admin] PATH_INFO: `/v3/projects/37e018b1953a48e29581746ca5a0cfc8/users/1fe70920c11a4dd8ad47de404eba50f3/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:44.269967 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-eae1df68-e291-44b9-812c-37a3bac0d26f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fb90ed7977fc4a7dabb4fbaff61fe95e'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:44.295367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605fb49a-4918-46ab-ad73-1f42c72e7374 admin admin] RBAC: Authorizing `identity:create_grant(project_id=37e018b1953a48e29581746ca5a0cfc8, user_id=1fe70920c11a4dd8ad47de404eba50f3, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:44.309479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-eae1df68-e291-44b9-812c-37a3bac0d26f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fb90ed7977fc4a7dabb4fbaff61fe95e'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:44.309479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-eae1df68-e291-44b9-812c-37a3bac0d26f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fb90ed7977fc4a7dabb4fbaff61fe95e'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:44.309823 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605fb49a-4918-46ab-ad73-1f42c72e7374 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:44.322090 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-eae1df68-e291-44b9-812c-37a3bac0d26f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fb90ed7977fc4a7dabb4fbaff61fe95e'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:44.329045 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 941/1860] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:43 2022] DELETE /identity/v3/users/fb90ed7977fc4a7dabb4fbaff61fe95e => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:44.339716 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c862dbbc-57cc-4cc7-b1ef-bb1c6b1603fb None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:44.340258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c862dbbc-57cc-4cc7-b1ef-bb1c6b1603fb None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:44.340632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c862dbbc-57cc-4cc7-b1ef-bb1c6b1603fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:44.356496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:44.369961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c862dbbc-57cc-4cc7-b1ef-bb1c6b1603fb None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:44.371354 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c862dbbc-57cc-4cc7-b1ef-bb1c6b1603fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:44.385227 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 920/1861] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:44 2022] PUT /identity/v3/projects/37e018b1953a48e29581746ca5a0cfc8/users/1fe70920c11a4dd8ad47de404eba50f3/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:44.389238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605fb49a-4918-46ab-ad73-1f42c72e7374 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:44.390839 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-605fb49a-4918-46ab-ad73-1f42c72e7374 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:44.479400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 942/1862] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:02:44.486908 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c862dbbc-57cc-4cc7-b1ef-bb1c6b1603fb None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:44.491303 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c862dbbc-57cc-4cc7-b1ef-bb1c6b1603fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:44.532398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605fb49a-4918-46ab-ad73-1f42c72e7374 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:44.537434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efef88c-ac32-49b7-9e29-58e3860e239c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:44.538208 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efef88c-ac32-49b7-9e29-58e3860e239c admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:44.538617 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efef88c-ac32-49b7-9e29-58e3860e239c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:44.539038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efef88c-ac32-49b7-9e29-58e3860e239c admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:44.539687 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7efef88c-ac32-49b7-9e29-58e3860e239c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:44.544335 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7efef88c-ac32-49b7-9e29-58e3860e239c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:44.562564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 921/1863] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:44 2022] GET /identity/v3/roles => generated 1420 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:02:44.572598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efef88c-ac32-49b7-9e29-58e3860e239c admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:44.572777 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c862dbbc-57cc-4cc7-b1ef-bb1c6b1603fb None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:44.577945 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efef88c-ac32-49b7-9e29-58e3860e239c admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:44.585170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccc0516-54c3-45a9-8851-34b955782c06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:44.586088 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccc0516-54c3-45a9-8851-34b955782c06 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:44.586535 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccc0516-54c3-45a9-8851-34b955782c06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:44.586914 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccc0516-54c3-45a9-8851-34b955782c06 admin admin] PATH_INFO: `/v3/projects/37e018b1953a48e29581746ca5a0cfc8/users/1fe70920c11a4dd8ad47de404eba50f3/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:44.589017 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ccc0516-54c3-45a9-8851-34b955782c06 admin admin] RBAC: Authorizing `identity:create_grant(project_id=37e018b1953a48e29581746ca5a0cfc8, user_id=1fe70920c11a4dd8ad47de404eba50f3, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:44.592124 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ccc0516-54c3-45a9-8851-34b955782c06 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:44.630911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 922/1864] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:44 2022] PUT /identity/v3/projects/37e018b1953a48e29581746ca5a0cfc8/users/1fe70920c11a4dd8ad47de404eba50f3/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:44.632654 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb7c174-f26b-4620-a682-d612822ed9f0 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:44.632941 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb7c174-f26b-4620-a682-d612822ed9f0 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:44.633209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb7c174-f26b-4620-a682-d612822ed9f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:44.668872 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a356aca-6e05-4e81-8bac-ea7e8a49726e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:44.669480 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a356aca-6e05-4e81-8bac-ea7e8a49726e service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:44.669754 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a356aca-6e05-4e81-8bac-ea7e8a49726e service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:44.670013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a356aca-6e05-4e81-8bac-ea7e8a49726e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:44.678997 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-4eb7c174-f26b-4620-a682-d612822ed9f0 None None] MFA Rules not processed for user `1fe70920c11a4dd8ad47de404eba50f3`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:44.680478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-4eb7c174-f26b-4620-a682-d612822ed9f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:44.681382 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a356aca-6e05-4e81-8bac-ea7e8a49726e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:44.682993 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a356aca-6e05-4e81-8bac-ea7e8a49726e service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:44.711968 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 943/1865] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:02:44.727896 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:44.851927 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 923/1866] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:44 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:44.864825 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a356aca-6e05-4e81-8bac-ea7e8a49726e service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:44.867233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-3a356aca-6e05-4e81-8bac-ea7e8a49726e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:44.975708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a356aca-6e05-4e81-8bac-ea7e8a49726e service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:45.025498 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=85,keystone:DELETE=11,keystone:INSERT=4 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:45.032663 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0380932f-023a-4a2e-9284-600eb62a5b7f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:45.033280 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0380932f-023a-4a2e-9284-600eb62a5b7f service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:45.033579 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0380932f-023a-4a2e-9284-600eb62a5b7f service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:45.033888 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0380932f-023a-4a2e-9284-600eb62a5b7f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:45.048050 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0380932f-023a-4a2e-9284-600eb62a5b7f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:45.049586 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0380932f-023a-4a2e-9284-600eb62a5b7f service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:45.067877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 944/1867] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:44 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:45.388476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb7c174-f26b-4620-a682-d612822ed9f0 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:45.672786 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-4eb7c174-f26b-4620-a682-d612822ed9f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:45.672786 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb7c174-f26b-4620-a682-d612822ed9f0 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:45.672786 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53404870-3ec0-4a05-b634-e43c08825586 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:45.675341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53404870-3ec0-4a05-b634-e43c08825586 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:45.675341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53404870-3ec0-4a05-b634-e43c08825586 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:45.675341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53404870-3ec0-4a05-b634-e43c08825586 admin admin] PATH_INFO: `/v3/projects/a33bc4bba7f5413285659421477c1081` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:45.675341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53404870-3ec0-4a05-b634-e43c08825586 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a33bc4bba7f5413285659421477c1081)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:45.675341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53404870-3ec0-4a05-b634-e43c08825586 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:45.821522 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-53404870-3ec0-4a05-b634-e43c08825586 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a33bc4bba7f5413285659421477c1081', 'request_id': 'req-53404870-3ec0-4a05-b634-e43c08825586'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:46.178905 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-53404870-3ec0-4a05-b634-e43c08825586 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a33bc4bba7f5413285659421477c1081', 'request_id': 'req-53404870-3ec0-4a05-b634-e43c08825586'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:46.186559 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-53404870-3ec0-4a05-b634-e43c08825586 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a33bc4bba7f5413285659421477c1081', 'request_id': 'req-53404870-3ec0-4a05-b634-e43c08825586'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:46.219687 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-53404870-3ec0-4a05-b634-e43c08825586 admin admin] The token cache is being invalidate because project a33bc4bba7f5413285659421477c1081 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:46.230423 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-53404870-3ec0-4a05-b634-e43c08825586 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:46.230423 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-53404870-3ec0-4a05-b634-e43c08825586 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:46.230423 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 924/1868] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:45 2022] DELETE /identity/v3/projects/a33bc4bba7f5413285659421477c1081 => generated 0 bytes in 836 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:02:46.234005 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0380932f-023a-4a2e-9284-600eb62a5b7f service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:46.247024 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0380932f-023a-4a2e-9284-600eb62a5b7f service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:46.306927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a9c20a-57c1-4c0e-9fc9-ef37cd982c4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:46.307558 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a9c20a-57c1-4c0e-9fc9-ef37cd982c4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:46.307818 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a9c20a-57c1-4c0e-9fc9-ef37cd982c4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:46.308210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a9c20a-57c1-4c0e-9fc9-ef37cd982c4d admin admin] PATH_INFO: `/v3/users/bee1fcf80afb40ab9a68f16184908885` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:46.309328 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a9c20a-57c1-4c0e-9fc9-ef37cd982c4d admin admin] RBAC: Authorizing `identity:delete_user(user_id=bee1fcf80afb40ab9a68f16184908885)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:46.311362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a9c20a-57c1-4c0e-9fc9-ef37cd982c4d admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:46.544765 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:46.560662 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-74a9c20a-57c1-4c0e-9fc9-ef37cd982c4d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bee1fcf80afb40ab9a68f16184908885'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:46.635425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-74a9c20a-57c1-4c0e-9fc9-ef37cd982c4d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bee1fcf80afb40ab9a68f16184908885'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:46.649826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-74a9c20a-57c1-4c0e-9fc9-ef37cd982c4d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bee1fcf80afb40ab9a68f16184908885'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:46.650592 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-74a9c20a-57c1-4c0e-9fc9-ef37cd982c4d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bee1fcf80afb40ab9a68f16184908885'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:46.657720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-74a9c20a-57c1-4c0e-9fc9-ef37cd982c4d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bee1fcf80afb40ab9a68f16184908885'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:46.668182 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 945/1869] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:46 2022] DELETE /identity/v3/users/bee1fcf80afb40ab9a68f16184908885 => generated 0 bytes in 431 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:46.866400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:47.186645 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53404870-3ec0-4a05-b634-e43c08825586 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:47.188081 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-53404870-3ec0-4a05-b634-e43c08825586 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:47.213141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53404870-3ec0-4a05-b634-e43c08825586 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:47.312641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b09c7f-8197-42f5-8c98-1af51aea81c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:47.313294 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b09c7f-8197-42f5-8c98-1af51aea81c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:47.313576 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b09c7f-8197-42f5-8c98-1af51aea81c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:47.313944 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b09c7f-8197-42f5-8c98-1af51aea81c4 admin admin] PATH_INFO: `/v3/projects/0e8fbbc2b5da4f769bfea42f5d94a625` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:47.321811 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b09c7f-8197-42f5-8c98-1af51aea81c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0e8fbbc2b5da4f769bfea42f5d94a625)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:47.323787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b09c7f-8197-42f5-8c98-1af51aea81c4 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:47.429721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f1b09c7f-8197-42f5-8c98-1af51aea81c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0e8fbbc2b5da4f769bfea42f5d94a625', 'request_id': 'req-f1b09c7f-8197-42f5-8c98-1af51aea81c4'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:47.503322 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f1b09c7f-8197-42f5-8c98-1af51aea81c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0e8fbbc2b5da4f769bfea42f5d94a625', 'request_id': 'req-f1b09c7f-8197-42f5-8c98-1af51aea81c4'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:47.509930 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f1b09c7f-8197-42f5-8c98-1af51aea81c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0e8fbbc2b5da4f769bfea42f5d94a625', 'request_id': 'req-f1b09c7f-8197-42f5-8c98-1af51aea81c4'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:47.522520 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f1b09c7f-8197-42f5-8c98-1af51aea81c4 admin admin] The token cache is being invalidate because project 0e8fbbc2b5da4f769bfea42f5d94a625 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:47.522814 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f1b09c7f-8197-42f5-8c98-1af51aea81c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:47.523490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f1b09c7f-8197-42f5-8c98-1af51aea81c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:47.525005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 925/1870] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:47 2022] DELETE /identity/v3/projects/0e8fbbc2b5da4f769bfea42f5d94a625 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:47.529854 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913164ab-5642-4c97-8500-a9c4561b5f30 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:47.530372 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913164ab-5642-4c97-8500-a9c4561b5f30 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:47.530475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913164ab-5642-4c97-8500-a9c4561b5f30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:47.532788 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:47.550310 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-913164ab-5642-4c97-8500-a9c4561b5f30 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:47.551693 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-913164ab-5642-4c97-8500-a9c4561b5f30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:47.615190 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:47.617889 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 946/1871] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:47.626885 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923c8d80-15b7-4f2b-befe-a3cba3537771 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:47.627170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923c8d80-15b7-4f2b-befe-a3cba3537771 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:47.627468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923c8d80-15b7-4f2b-befe-a3cba3537771 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:47.648085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-923c8d80-15b7-4f2b-befe-a3cba3537771 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:47.649317 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-923c8d80-15b7-4f2b-befe-a3cba3537771 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:47.657160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 926/1872] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:47.661329 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913164ab-5642-4c97-8500-a9c4561b5f30 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:47.670953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913164ab-5642-4c97-8500-a9c4561b5f30 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:47.675873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a806b6fc-e225-4930-bb71-8ff31e2c5819 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:47.676491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a806b6fc-e225-4930-bb71-8ff31e2c5819 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:47.676770 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a806b6fc-e225-4930-bb71-8ff31e2c5819 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:47.677037 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a806b6fc-e225-4930-bb71-8ff31e2c5819 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:47.677939 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a806b6fc-e225-4930-bb71-8ff31e2c5819 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:47.678238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a806b6fc-e225-4930-bb71-8ff31e2c5819 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:47.679866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a806b6fc-e225-4930-bb71-8ff31e2c5819 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:47.699163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 947/1873] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:02:47.700432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923c8d80-15b7-4f2b-befe-a3cba3537771 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:47.710648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923c8d80-15b7-4f2b-befe-a3cba3537771 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:47.715709 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07eae488-df76-4382-8664-52de29d38694 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:47.716326 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eae488-df76-4382-8664-52de29d38694 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:47.716606 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eae488-df76-4382-8664-52de29d38694 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:47.716879 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eae488-df76-4382-8664-52de29d38694 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:47.717578 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07eae488-df76-4382-8664-52de29d38694 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:47.719548 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07eae488-df76-4382-8664-52de29d38694 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:47.814083 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 927/1874] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:47 2022] POST /identity/v3/projects => generated 383 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:02:47.815342 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a806b6fc-e225-4930-bb71-8ff31e2c5819 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:47.829666 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a806b6fc-e225-4930-bb71-8ff31e2c5819 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:47.855610 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:47.922727 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a123559-8e3a-41d7-af3c-728096ac8a0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:47.923454 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a123559-8e3a-41d7-af3c-728096ac8a0b admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:47.923612 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a123559-8e3a-41d7-af3c-728096ac8a0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:47.924051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a123559-8e3a-41d7-af3c-728096ac8a0b admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:47.924737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a123559-8e3a-41d7-af3c-728096ac8a0b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:47.928013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a123559-8e3a-41d7-af3c-728096ac8a0b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:48.040745 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 948/1875] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:47 2022] POST /identity/v3/users => generated 292 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:48.041801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07eae488-df76-4382-8664-52de29d38694 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:48.058750 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07eae488-df76-4382-8664-52de29d38694 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:48.063378 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9a1133-62fa-4d2c-bdb8-112c3400b1dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:48.063981 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9a1133-62fa-4d2c-bdb8-112c3400b1dd admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:48.064253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9a1133-62fa-4d2c-bdb8-112c3400b1dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:48.064539 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9a1133-62fa-4d2c-bdb8-112c3400b1dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:48.065146 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9a1133-62fa-4d2c-bdb8-112c3400b1dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:48.067528 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9a1133-62fa-4d2c-bdb8-112c3400b1dd admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:48.079684 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 928/1876] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:48 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:48.092491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a123559-8e3a-41d7-af3c-728096ac8a0b admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:48.103665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a123559-8e3a-41d7-af3c-728096ac8a0b admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:48.108408 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a789979d-f106-490e-a048-c0931d3a212f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:48.113565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a789979d-f106-490e-a048-c0931d3a212f admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:48.113565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a789979d-f106-490e-a048-c0931d3a212f admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:48.113565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a789979d-f106-490e-a048-c0931d3a212f admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:48.113565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a789979d-f106-490e-a048-c0931d3a212f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:48.113565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a789979d-f106-490e-a048-c0931d3a212f admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:48.125400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 949/1877] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:48 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:48.130470 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9a1133-62fa-4d2c-bdb8-112c3400b1dd admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:48.143817 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9a1133-62fa-4d2c-bdb8-112c3400b1dd admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:48.148551 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c398820e-43b3-46d0-ac97-19f1bcefb114 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:48.149212 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398820e-43b3-46d0-ac97-19f1bcefb114 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:48.149490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398820e-43b3-46d0-ac97-19f1bcefb114 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:48.149852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c398820e-43b3-46d0-ac97-19f1bcefb114 admin admin] PATH_INFO: `/v3/projects/8e7ff127ca7146df872525477b42f11d/users/d8a908a871c94799b96874aca602e984/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:48.170232 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c398820e-43b3-46d0-ac97-19f1bcefb114 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e7ff127ca7146df872525477b42f11d, user_id=d8a908a871c94799b96874aca602e984, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:48.170232 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c398820e-43b3-46d0-ac97-19f1bcefb114 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:48.271036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 929/1878] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:48 2022] PUT /identity/v3/projects/8e7ff127ca7146df872525477b42f11d/users/d8a908a871c94799b96874aca602e984/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:48.282165 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3aafe4-e782-4a08-860e-e958fbc809d6 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:48.282165 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3aafe4-e782-4a08-860e-e958fbc809d6 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:48.282427 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3aafe4-e782-4a08-860e-e958fbc809d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:48.340912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:48.340912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-ac3aafe4-e782-4a08-860e-e958fbc809d6 None None] MFA Rules not processed for user `d8a908a871c94799b96874aca602e984`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:48.341436 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-ac3aafe4-e782-4a08-860e-e958fbc809d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:48.436848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:48.444483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 950/1879] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:48 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:48.448329 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c398820e-43b3-46d0-ac97-19f1bcefb114 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:48.536776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c398820e-43b3-46d0-ac97-19f1bcefb114 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:48.542794 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be90bd8c-6976-4137-99f5-abb5d400ee5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:48.544311 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be90bd8c-6976-4137-99f5-abb5d400ee5b admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:48.544588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be90bd8c-6976-4137-99f5-abb5d400ee5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:48.545027 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be90bd8c-6976-4137-99f5-abb5d400ee5b admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:48.545827 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be90bd8c-6976-4137-99f5-abb5d400ee5b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:48.548117 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be90bd8c-6976-4137-99f5-abb5d400ee5b admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:48.670233 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 930/1880] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:48 2022] POST /identity/v3/projects => generated 383 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:48.678695 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3aafe4-e782-4a08-860e-e958fbc809d6 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:48.691713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3aafe4-e782-4a08-860e-e958fbc809d6 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:48.779209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387f26bf-3973-4b2a-b669-0b0cf31d45c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:48.779938 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387f26bf-3973-4b2a-b669-0b0cf31d45c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:48.780342 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387f26bf-3973-4b2a-b669-0b0cf31d45c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:48.780710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387f26bf-3973-4b2a-b669-0b0cf31d45c7 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:48.781603 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387f26bf-3973-4b2a-b669-0b0cf31d45c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:48.783721 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387f26bf-3973-4b2a-b669-0b0cf31d45c7 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:48.886651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 951/1881] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:48 2022] POST /identity/v3/users => generated 298 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:02:48.888994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be90bd8c-6976-4137-99f5-abb5d400ee5b admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:48.901929 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be90bd8c-6976-4137-99f5-abb5d400ee5b admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:48.910777 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4105fd28-46a4-4fc1-ba5c-ca5a36c494a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:48.915844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4105fd28-46a4-4fc1-ba5c-ca5a36c494a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:48.915844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4105fd28-46a4-4fc1-ba5c-ca5a36c494a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:48.915844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4105fd28-46a4-4fc1-ba5c-ca5a36c494a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:48.915844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4105fd28-46a4-4fc1-ba5c-ca5a36c494a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:48.915844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4105fd28-46a4-4fc1-ba5c-ca5a36c494a7 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:48.928119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 931/1882] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:48 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:48.932373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387f26bf-3973-4b2a-b669-0b0cf31d45c7 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:48.947232 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387f26bf-3973-4b2a-b669-0b0cf31d45c7 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:48.953547 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f7212f-7be1-460e-9df0-06f1cf42510b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:48.954624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f7212f-7be1-460e-9df0-06f1cf42510b admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:48.954903 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f7212f-7be1-460e-9df0-06f1cf42510b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:48.955475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f7212f-7be1-460e-9df0-06f1cf42510b admin admin] PATH_INFO: `/v3/projects/50b01a27686b416f90747fa4f4f80a9b/users/875446d2354e44a18b8db47188ac9b48/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:48.973881 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f7212f-7be1-460e-9df0-06f1cf42510b admin admin] RBAC: Authorizing `identity:create_grant(project_id=50b01a27686b416f90747fa4f4f80a9b, user_id=875446d2354e44a18b8db47188ac9b48, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:48.976921 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f7212f-7be1-460e-9df0-06f1cf42510b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:49.048455 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 952/1883] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:48 2022] PUT /identity/v3/projects/50b01a27686b416f90747fa4f4f80a9b/users/875446d2354e44a18b8db47188ac9b48/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:49.052836 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4105fd28-46a4-4fc1-ba5c-ca5a36c494a7 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:49.053588 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:49.065860 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4105fd28-46a4-4fc1-ba5c-ca5a36c494a7 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:49.101380 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:49.152046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4ab7d2-5d9e-4ed3-8820-e690eaf4f3d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:49.153044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4ab7d2-5d9e-4ed3-8820-e690eaf4f3d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:49.153527 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4ab7d2-5d9e-4ed3-8820-e690eaf4f3d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:49.153878 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4ab7d2-5d9e-4ed3-8820-e690eaf4f3d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:49.154638 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4ab7d2-5d9e-4ed3-8820-e690eaf4f3d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:49.155987 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f7212f-7be1-460e-9df0-06f1cf42510b admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:49.157876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-e7f7212f-7be1-460e-9df0-06f1cf42510b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:49.158047 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4ab7d2-5d9e-4ed3-8820-e690eaf4f3d7 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:49.170402 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 932/1884] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:49 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:49.174115 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4ab7d2-5d9e-4ed3-8820-e690eaf4f3d7 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:49.179602 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f7212f-7be1-460e-9df0-06f1cf42510b admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:49.187071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f4c4bc-432c-45c0-ad99-06e428b077d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:49.187699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4ab7d2-5d9e-4ed3-8820-e690eaf4f3d7 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:49.187821 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f4c4bc-432c-45c0-ad99-06e428b077d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:49.188160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f4c4bc-432c-45c0-ad99-06e428b077d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:49.189306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f4c4bc-432c-45c0-ad99-06e428b077d4 admin admin] PATH_INFO: `/v3/users/14c177b6fe5642d7aff1bd8bc049107a` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:49.189934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f4c4bc-432c-45c0-ad99-06e428b077d4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=14c177b6fe5642d7aff1bd8bc049107a)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:49.192841 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f4c4bc-432c-45c0-ad99-06e428b077d4 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:49.215202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b8bfa8-0121-45f5-bcec-212a6045f22e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:49.215202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b8bfa8-0121-45f5-bcec-212a6045f22e admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:49.215202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b8bfa8-0121-45f5-bcec-212a6045f22e admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:49.215202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b8bfa8-0121-45f5-bcec-212a6045f22e admin admin] PATH_INFO: `/v3/projects/50b01a27686b416f90747fa4f4f80a9b/users/875446d2354e44a18b8db47188ac9b48/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:49.220797 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b8bfa8-0121-45f5-bcec-212a6045f22e admin admin] RBAC: Authorizing `identity:create_grant(project_id=50b01a27686b416f90747fa4f4f80a9b, user_id=875446d2354e44a18b8db47188ac9b48, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:49.222777 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b8bfa8-0121-45f5-bcec-212a6045f22e admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:49.233795 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 933/1885] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:49 2022] PUT /identity/v3/projects/50b01a27686b416f90747fa4f4f80a9b/users/875446d2354e44a18b8db47188ac9b48/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:02:49.244713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6266fe4d-63ff-4665-82bc-56f7caeca264 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:49.245127 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6266fe4d-63ff-4665-82bc-56f7caeca264 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:49.245505 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6266fe4d-63ff-4665-82bc-56f7caeca264 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:49.285864 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-6266fe4d-63ff-4665-82bc-56f7caeca264 None None] MFA Rules not processed for user `875446d2354e44a18b8db47188ac9b48`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:49.287894 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-6266fe4d-63ff-4665-82bc-56f7caeca264 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:49.367777 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 934/1886] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:49 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:02:49.403780 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:49.418849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-73f4c4bc-432c-45c0-ad99-06e428b077d4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '14c177b6fe5642d7aff1bd8bc049107a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:49.467510 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6266fe4d-63ff-4665-82bc-56f7caeca264 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:49.468907 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-6266fe4d-63ff-4665-82bc-56f7caeca264 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:49.543947 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-73f4c4bc-432c-45c0-ad99-06e428b077d4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '14c177b6fe5642d7aff1bd8bc049107a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:49.556588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6266fe4d-63ff-4665-82bc-56f7caeca264 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:49.561424 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67a7021-ab6c-4e0d-abfa-b4cb4ed98fd3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:49.562995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67a7021-ab6c-4e0d-abfa-b4cb4ed98fd3 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:49.563527 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67a7021-ab6c-4e0d-abfa-b4cb4ed98fd3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:49.564097 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67a7021-ab6c-4e0d-abfa-b4cb4ed98fd3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:49.564446 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-73f4c4bc-432c-45c0-ad99-06e428b077d4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '14c177b6fe5642d7aff1bd8bc049107a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:49.565307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-73f4c4bc-432c-45c0-ad99-06e428b077d4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '14c177b6fe5642d7aff1bd8bc049107a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:49.567358 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a67a7021-ab6c-4e0d-abfa-b4cb4ed98fd3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:49.574930 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-73f4c4bc-432c-45c0-ad99-06e428b077d4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '14c177b6fe5642d7aff1bd8bc049107a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:49.586820 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 953/1887] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:49 2022] DELETE /identity/v3/users/14c177b6fe5642d7aff1bd8bc049107a => generated 0 bytes in 427 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:02:49.587484 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5542db-478b-4fc5-83eb-69f1417fd561 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:49.587909 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5542db-478b-4fc5-83eb-69f1417fd561 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:49.588231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5542db-478b-4fc5-83eb-69f1417fd561 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:49.593200 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67a7021-ab6c-4e0d-abfa-b4cb4ed98fd3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:49.595695 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67a7021-ab6c-4e0d-abfa-b4cb4ed98fd3 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:49.608283 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 935/1888] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:49 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:49.619515 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-6b5542db-478b-4fc5-83eb-69f1417fd561 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:49.620723 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-6b5542db-478b-4fc5-83eb-69f1417fd561 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:49.705860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 954/1889] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:49.725917 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67a7021-ab6c-4e0d-abfa-b4cb4ed98fd3 service cinder] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:49.730663 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a67a7021-ab6c-4e0d-abfa-b4cb4ed98fd3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:49.759362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5542db-478b-4fc5-83eb-69f1417fd561 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:49.761763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-6b5542db-478b-4fc5-83eb-69f1417fd561 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:49.802614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67a7021-ab6c-4e0d-abfa-b4cb4ed98fd3 service cinder] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:49.829641 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5542db-478b-4fc5-83eb-69f1417fd561 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:49.834481 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66f8cc7-a0bd-4977-bf12-bca1472b6eaf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:49.835133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66f8cc7-a0bd-4977-bf12-bca1472b6eaf service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:49.835415 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66f8cc7-a0bd-4977-bf12-bca1472b6eaf service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:49.835687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66f8cc7-a0bd-4977-bf12-bca1472b6eaf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:49.838559 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d66f8cc7-a0bd-4977-bf12-bca1472b6eaf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:49.865532 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e10ee9-ca8f-47ce-a4dd-f7281d93fba5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:49.866208 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e10ee9-ca8f-47ce-a4dd-f7281d93fba5 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:49.866808 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e10ee9-ca8f-47ce-a4dd-f7281d93fba5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:49.866944 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e10ee9-ca8f-47ce-a4dd-f7281d93fba5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:49.881527 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e10ee9-ca8f-47ce-a4dd-f7281d93fba5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:49.884323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e10ee9-ca8f-47ce-a4dd-f7281d93fba5 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:49.911448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 936/1890] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:49.936885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d66f8cc7-a0bd-4977-bf12-bca1472b6eaf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:49.936885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d66f8cc7-a0bd-4977-bf12-bca1472b6eaf service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:49.954684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 955/1891] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:49 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 195 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:50.576124 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e10ee9-ca8f-47ce-a4dd-f7281d93fba5 service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:50.578071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f5e10ee9-ca8f-47ce-a4dd-f7281d93fba5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:50.593767 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e10ee9-ca8f-47ce-a4dd-f7281d93fba5 service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:50.602392 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869a22f8-05d7-447a-b3a4-1acad8258ce6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:50.603078 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869a22f8-05d7-447a-b3a4-1acad8258ce6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:50.603361 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869a22f8-05d7-447a-b3a4-1acad8258ce6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:50.603634 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869a22f8-05d7-447a-b3a4-1acad8258ce6 admin admin] PATH_INFO: `/v3/projects/607dd350513840c7a6476644388b4a08` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:50.604975 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869a22f8-05d7-447a-b3a4-1acad8258ce6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=607dd350513840c7a6476644388b4a08)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:50.607222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869a22f8-05d7-447a-b3a4-1acad8258ce6 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:50.740496 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-869a22f8-05d7-447a-b3a4-1acad8258ce6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '607dd350513840c7a6476644388b4a08', 'request_id': 'req-869a22f8-05d7-447a-b3a4-1acad8258ce6'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:50.835811 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-869a22f8-05d7-447a-b3a4-1acad8258ce6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '607dd350513840c7a6476644388b4a08', 'request_id': 'req-869a22f8-05d7-447a-b3a4-1acad8258ce6'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:50.841883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-869a22f8-05d7-447a-b3a4-1acad8258ce6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '607dd350513840c7a6476644388b4a08', 'request_id': 'req-869a22f8-05d7-447a-b3a4-1acad8258ce6'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:50.853544 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-869a22f8-05d7-447a-b3a4-1acad8258ce6 admin admin] The token cache is being invalidate because project 607dd350513840c7a6476644388b4a08 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:50.853772 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-869a22f8-05d7-447a-b3a4-1acad8258ce6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:50.854486 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-869a22f8-05d7-447a-b3a4-1acad8258ce6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:50.855976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 937/1892] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:50 2022] DELETE /identity/v3/projects/607dd350513840c7a6476644388b4a08 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:50.859977 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66f8cc7-a0bd-4977-bf12-bca1472b6eaf service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:50.872444 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66f8cc7-a0bd-4977-bf12-bca1472b6eaf service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:50.933346 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286b6fcc-4d44-4e75-9972-74ce0a696050 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:50.933985 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286b6fcc-4d44-4e75-9972-74ce0a696050 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:50.934268 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286b6fcc-4d44-4e75-9972-74ce0a696050 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:50.934575 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286b6fcc-4d44-4e75-9972-74ce0a696050 admin admin] PATH_INFO: `/v3/users/1fe70920c11a4dd8ad47de404eba50f3` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:50.935711 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286b6fcc-4d44-4e75-9972-74ce0a696050 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1fe70920c11a4dd8ad47de404eba50f3)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:50.937637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286b6fcc-4d44-4e75-9972-74ce0a696050 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:51.441361 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:52.256307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-286b6fcc-4d44-4e75-9972-74ce0a696050 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1fe70920c11a4dd8ad47de404eba50f3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:52.360924 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-286b6fcc-4d44-4e75-9972-74ce0a696050 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1fe70920c11a4dd8ad47de404eba50f3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:52.375915 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-286b6fcc-4d44-4e75-9972-74ce0a696050 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1fe70920c11a4dd8ad47de404eba50f3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:52.376748 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-286b6fcc-4d44-4e75-9972-74ce0a696050 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1fe70920c11a4dd8ad47de404eba50f3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:52.383508 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-286b6fcc-4d44-4e75-9972-74ce0a696050 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1fe70920c11a4dd8ad47de404eba50f3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:52.389537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 956/1893] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:50 2022] DELETE /identity/v3/users/1fe70920c11a4dd8ad47de404eba50f3 => generated 0 bytes in 1530 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:52.396118 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:52.891253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869a22f8-05d7-447a-b3a4-1acad8258ce6 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:52.892777 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-869a22f8-05d7-447a-b3a4-1acad8258ce6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:52.911710 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869a22f8-05d7-447a-b3a4-1acad8258ce6 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:53.013561 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b70bc0-66ff-4973-81d4-f4f5043b3eda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:53.014754 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b70bc0-66ff-4973-81d4-f4f5043b3eda admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:53.039904 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b70bc0-66ff-4973-81d4-f4f5043b3eda admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:53.039904 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b70bc0-66ff-4973-81d4-f4f5043b3eda admin admin] PATH_INFO: `/v3/projects/37e018b1953a48e29581746ca5a0cfc8` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:53.041286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b70bc0-66ff-4973-81d4-f4f5043b3eda admin admin] RBAC: Authorizing `identity:delete_project(project_id=37e018b1953a48e29581746ca5a0cfc8)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:53.043273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b70bc0-66ff-4973-81d4-f4f5043b3eda admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:53.227520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef75d9f1-a6b2-4842-9b10-4ad25119aed3 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:53.228713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef75d9f1-a6b2-4842-9b10-4ad25119aed3 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:53.229363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef75d9f1-a6b2-4842-9b10-4ad25119aed3 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:53.231659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 957/1894] 10.222.0.62 () {54 vars in 921 bytes} [Fri Sep 23 14:02:53 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:02:53.243002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-91b70bc0-66ff-4973-81d4-f4f5043b3eda admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '37e018b1953a48e29581746ca5a0cfc8', 'request_id': 'req-91b70bc0-66ff-4973-81d4-f4f5043b3eda'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:53.262528 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df87431-d159-475c-8d4c-a33b182a9332 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:53.263888 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df87431-d159-475c-8d4c-a33b182a9332 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:53.264383 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df87431-d159-475c-8d4c-a33b182a9332 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:53.293425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-2df87431-d159-475c-8d4c-a33b182a9332 None None] MFA Rules not processed for user `f01ed94dbb384a178ad0a45a6f3d8c17`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:53.295304 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-2df87431-d159-475c-8d4c-a33b182a9332 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:53.395852 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 958/1895] 10.222.0.62 () {58 vars in 1018 bytes} [Fri Sep 23 14:02:53 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:53.415307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df87431-d159-475c-8d4c-a33b182a9332 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:53.416907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-2df87431-d159-475c-8d4c-a33b182a9332 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:53.431817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df87431-d159-475c-8d4c-a33b182a9332 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:53.438044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030dba6c-e9ff-4e3e-bbbb-bb578c6ddd12 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:53.438710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030dba6c-e9ff-4e3e-bbbb-bb578c6ddd12 service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:53.439069 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030dba6c-e9ff-4e3e-bbbb-bb578c6ddd12 service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:53.439315 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030dba6c-e9ff-4e3e-bbbb-bb578c6ddd12 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:53.449641 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030dba6c-e9ff-4e3e-bbbb-bb578c6ddd12 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:53.451167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030dba6c-e9ff-4e3e-bbbb-bb578c6ddd12 service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:53.458771 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-91b70bc0-66ff-4973-81d4-f4f5043b3eda admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '37e018b1953a48e29581746ca5a0cfc8', 'request_id': 'req-91b70bc0-66ff-4973-81d4-f4f5043b3eda'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:53.463237 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 959/1896] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:53.465663 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-91b70bc0-66ff-4973-81d4-f4f5043b3eda admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '37e018b1953a48e29581746ca5a0cfc8', 'request_id': 'req-91b70bc0-66ff-4973-81d4-f4f5043b3eda'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:53.478989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-91b70bc0-66ff-4973-81d4-f4f5043b3eda admin admin] The token cache is being invalidate because project 37e018b1953a48e29581746ca5a0cfc8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:53.479438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-91b70bc0-66ff-4973-81d4-f4f5043b3eda admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:53.480184 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-91b70bc0-66ff-4973-81d4-f4f5043b3eda admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:53.482113 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 938/1897] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:52 2022] DELETE /identity/v3/projects/37e018b1953a48e29581746ca5a0cfc8 => generated 0 bytes in 591 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:53.487361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa3747e-4453-49da-be85-02407b75e3e6 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:53.487880 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa3747e-4453-49da-be85-02407b75e3e6 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:53.488248 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa3747e-4453-49da-be85-02407b75e3e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:53.512653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-bfa3747e-4453-49da-be85-02407b75e3e6 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:53.514013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-bfa3747e-4453-49da-be85-02407b75e3e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:53.624866 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:53.632810 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 960/1898] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:53.636901 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06645253-cd4a-49d3-b255-71b4fc37c821 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:53.637224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06645253-cd4a-49d3-b255-71b4fc37c821 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:53.637488 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06645253-cd4a-49d3-b255-71b4fc37c821 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:53.669827 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-06645253-cd4a-49d3-b255-71b4fc37c821 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:53.671108 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-06645253-cd4a-49d3-b255-71b4fc37c821 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:53.680486 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 939/1899] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:02:53.685059 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa3747e-4453-49da-be85-02407b75e3e6 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:53.695120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa3747e-4453-49da-be85-02407b75e3e6 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:53.700285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0927464b-fe0d-4f8d-beec-e080a4688149 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:53.700937 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0927464b-fe0d-4f8d-beec-e080a4688149 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:53.704685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0927464b-fe0d-4f8d-beec-e080a4688149 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:53.704685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0927464b-fe0d-4f8d-beec-e080a4688149 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:53.704685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0927464b-fe0d-4f8d-beec-e080a4688149 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:53.704685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0927464b-fe0d-4f8d-beec-e080a4688149 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:53.704685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0927464b-fe0d-4f8d-beec-e080a4688149 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:53.716596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 961/1900] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:02:53.722506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06645253-cd4a-49d3-b255-71b4fc37c821 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:53.730746 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06645253-cd4a-49d3-b255-71b4fc37c821 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:53.735607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23df3f7b-a021-4674-b1cd-72be269a186e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:53.736212 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23df3f7b-a021-4674-b1cd-72be269a186e admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:53.738891 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23df3f7b-a021-4674-b1cd-72be269a186e admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:53.738891 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23df3f7b-a021-4674-b1cd-72be269a186e admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:53.738891 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23df3f7b-a021-4674-b1cd-72be269a186e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:53.740311 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23df3f7b-a021-4674-b1cd-72be269a186e admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:53.835196 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:53.846045 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 940/1901] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:53 2022] POST /identity/v3/projects => generated 367 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:53.849683 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0927464b-fe0d-4f8d-beec-e080a4688149 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:53.860098 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0927464b-fe0d-4f8d-beec-e080a4688149 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:53.919547 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8d1e5e-7b33-4ba0-8892-a2b74d530e67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:53.920152 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8d1e5e-7b33-4ba0-8892-a2b74d530e67 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:53.920433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8d1e5e-7b33-4ba0-8892-a2b74d530e67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:53.920709 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8d1e5e-7b33-4ba0-8892-a2b74d530e67 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:53.921371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8d1e5e-7b33-4ba0-8892-a2b74d530e67 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:53.923341 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8d1e5e-7b33-4ba0-8892-a2b74d530e67 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:54.030050 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 962/1902] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:53 2022] POST /identity/v3/users => generated 284 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:54.034312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23df3f7b-a021-4674-b1cd-72be269a186e admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:54.046393 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23df3f7b-a021-4674-b1cd-72be269a186e admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:54.059255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe862992-62e6-468e-9585-c9f329581e48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:54.059255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe862992-62e6-468e-9585-c9f329581e48 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:54.059481 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe862992-62e6-468e-9585-c9f329581e48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:54.059717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe862992-62e6-468e-9585-c9f329581e48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:54.060965 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe862992-62e6-468e-9585-c9f329581e48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:54.062869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe862992-62e6-468e-9585-c9f329581e48 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:54.077034 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 941/1903] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:54 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:02:54.080991 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8d1e5e-7b33-4ba0-8892-a2b74d530e67 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:54.094076 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8d1e5e-7b33-4ba0-8892-a2b74d530e67 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:54.097517 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe862992-62e6-468e-9585-c9f329581e48 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:54.099375 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-fe862992-62e6-468e-9585-c9f329581e48 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:54.105342 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa52f0c9-3530-431f-bb8d-919cb3957f16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:54.105342 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa52f0c9-3530-431f-bb8d-919cb3957f16 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:54.105342 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa52f0c9-3530-431f-bb8d-919cb3957f16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:54.105342 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa52f0c9-3530-431f-bb8d-919cb3957f16 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:54.105723 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa52f0c9-3530-431f-bb8d-919cb3957f16 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:54.105723 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa52f0c9-3530-431f-bb8d-919cb3957f16 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:54.119488 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe862992-62e6-468e-9585-c9f329581e48 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:54.125479 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7c0bb9-e323-4bff-aa1e-ef23a1c00ee6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:54.126058 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 963/1904] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:54 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:54.129621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7c0bb9-e323-4bff-aa1e-ef23a1c00ee6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:54.130047 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7c0bb9-e323-4bff-aa1e-ef23a1c00ee6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:54.130433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7c0bb9-e323-4bff-aa1e-ef23a1c00ee6 admin admin] PATH_INFO: `/v3/users/242bf7d8bba64480814d8194ea12b19a` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:54.130986 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa52f0c9-3530-431f-bb8d-919cb3957f16 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:54.131884 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7c0bb9-e323-4bff-aa1e-ef23a1c00ee6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=242bf7d8bba64480814d8194ea12b19a)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:54.134183 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7c0bb9-e323-4bff-aa1e-ef23a1c00ee6 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:54.145430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa52f0c9-3530-431f-bb8d-919cb3957f16 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:54.151001 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76a56cc-038f-4bf9-bad4-145e76dfbaed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:54.151810 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76a56cc-038f-4bf9-bad4-145e76dfbaed admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:54.152184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76a56cc-038f-4bf9-bad4-145e76dfbaed admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:54.152778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76a56cc-038f-4bf9-bad4-145e76dfbaed admin admin] PATH_INFO: `/v3/projects/fdc51b455c8b418e81449c8488848521/users/f294eecaf0ee4fbf9f70a32780f950dd/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:54.174003 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76a56cc-038f-4bf9-bad4-145e76dfbaed admin admin] RBAC: Authorizing `identity:create_grant(project_id=fdc51b455c8b418e81449c8488848521, user_id=f294eecaf0ee4fbf9f70a32780f950dd, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:54.177149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76a56cc-038f-4bf9-bad4-145e76dfbaed admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:54.246884 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 964/1905] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:54 2022] PUT /identity/v3/projects/fdc51b455c8b418e81449c8488848521/users/f294eecaf0ee4fbf9f70a32780f950dd/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:02:54.258653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8966565a-511b-4642-b214-4c4fae11dab1 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:54.258934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8966565a-511b-4642-b214-4c4fae11dab1 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:54.259206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8966565a-511b-4642-b214-4c4fae11dab1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:54.287872 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-8966565a-511b-4642-b214-4c4fae11dab1 None None] MFA Rules not processed for user `f294eecaf0ee4fbf9f70a32780f950dd`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:54.289135 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8966565a-511b-4642-b214-4c4fae11dab1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:54.348880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-bc7c0bb9-e323-4bff-aa1e-ef23a1c00ee6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '242bf7d8bba64480814d8194ea12b19a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:54.357447 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 965/1906] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:54 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:02:54.362036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8966565a-511b-4642-b214-4c4fae11dab1 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:54.377907 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:54.413271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-bc7c0bb9-e323-4bff-aa1e-ef23a1c00ee6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '242bf7d8bba64480814d8194ea12b19a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:54.416122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-bc7c0bb9-e323-4bff-aa1e-ef23a1c00ee6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '242bf7d8bba64480814d8194ea12b19a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:54.434537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-bc7c0bb9-e323-4bff-aa1e-ef23a1c00ee6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '242bf7d8bba64480814d8194ea12b19a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:54.444189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8966565a-511b-4642-b214-4c4fae11dab1 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:54.450154 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791431ed-aeb3-41ce-ba84-6dc81ddecc92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:54.450761 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-bc7c0bb9-e323-4bff-aa1e-ef23a1c00ee6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '242bf7d8bba64480814d8194ea12b19a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:54.450881 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791431ed-aeb3-41ce-ba84-6dc81ddecc92 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:54.451692 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791431ed-aeb3-41ce-ba84-6dc81ddecc92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:54.452073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-791431ed-aeb3-41ce-ba84-6dc81ddecc92 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:54.453013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791431ed-aeb3-41ce-ba84-6dc81ddecc92 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:54.455202 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-791431ed-aeb3-41ce-ba84-6dc81ddecc92 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:54.461714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 942/1907] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:54 2022] DELETE /identity/v3/users/242bf7d8bba64480814d8194ea12b19a => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:02:54.466320 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cf7df7-222a-445e-b468-2d1017d5bb2c None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:54.466765 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cf7df7-222a-445e-b468-2d1017d5bb2c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:54.467144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cf7df7-222a-445e-b468-2d1017d5bb2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:54.488782 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-d1cf7df7-222a-445e-b468-2d1017d5bb2c None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:54.490195 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d1cf7df7-222a-445e-b468-2d1017d5bb2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:54.560146 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 943/1908] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:54.576119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cf7df7-222a-445e-b468-2d1017d5bb2c None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:54.582024 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 966/1909] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:54 2022] POST /identity/v3/projects => generated 367 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:02:54.582924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d1cf7df7-222a-445e-b468-2d1017d5bb2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:54.583195 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791431ed-aeb3-41ce-ba84-6dc81ddecc92 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:54.585017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-791431ed-aeb3-41ce-ba84-6dc81ddecc92 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:54.668178 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cf7df7-222a-445e-b468-2d1017d5bb2c None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:54.670696 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-791431ed-aeb3-41ce-ba84-6dc81ddecc92 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:54.675975 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4afbe6-4674-45db-a6e6-31719cbd5e06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:54.676792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4afbe6-4674-45db-a6e6-31719cbd5e06 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:54.677472 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4afbe6-4674-45db-a6e6-31719cbd5e06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:54.677934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4afbe6-4674-45db-a6e6-31719cbd5e06 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:54.678786 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4afbe6-4674-45db-a6e6-31719cbd5e06 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:54.680882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4afbe6-4674-45db-a6e6-31719cbd5e06 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:54.758197 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58539689-ee15-493b-8cdc-7d65f211d017 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:54.760459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58539689-ee15-493b-8cdc-7d65f211d017 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:54.761495 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58539689-ee15-493b-8cdc-7d65f211d017 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:54.761945 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58539689-ee15-493b-8cdc-7d65f211d017 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:54.779720 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58539689-ee15-493b-8cdc-7d65f211d017 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:54.781320 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58539689-ee15-493b-8cdc-7d65f211d017 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:54.792126 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 967/1910] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:54 2022] POST /identity/v3/users => generated 290 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:54.792126 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4afbe6-4674-45db-a6e6-31719cbd5e06 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:54.798419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 944/1911] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:54.818871 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4afbe6-4674-45db-a6e6-31719cbd5e06 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:54.827433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad278afa-71f3-4230-b04b-2e1521a59c28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:54.828405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad278afa-71f3-4230-b04b-2e1521a59c28 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:54.828962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad278afa-71f3-4230-b04b-2e1521a59c28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:54.829384 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad278afa-71f3-4230-b04b-2e1521a59c28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:54.830154 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad278afa-71f3-4230-b04b-2e1521a59c28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:54.832089 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad278afa-71f3-4230-b04b-2e1521a59c28 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:54.845368 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 968/1912] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:54 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:54.868069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58539689-ee15-493b-8cdc-7d65f211d017 service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:54.880642 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58539689-ee15-493b-8cdc-7d65f211d017 service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:54.900386 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291e3680-2e68-46dd-8fc8-bb330341dfc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:54.900386 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291e3680-2e68-46dd-8fc8-bb330341dfc3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:54.900386 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291e3680-2e68-46dd-8fc8-bb330341dfc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:54.900775 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291e3680-2e68-46dd-8fc8-bb330341dfc3 admin admin] PATH_INFO: `/v3/projects/0ccb60fdca144211a380fd193c8034b1/users/05fc5bc8a32042608a32cafe4de085bd/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:54.929686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-291e3680-2e68-46dd-8fc8-bb330341dfc3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ccb60fdca144211a380fd193c8034b1, user_id=05fc5bc8a32042608a32cafe4de085bd, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:54.932429 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-291e3680-2e68-46dd-8fc8-bb330341dfc3 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:54.984931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad278afa-71f3-4230-b04b-2e1521a59c28 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:54.992862 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-ad278afa-71f3-4230-b04b-2e1521a59c28 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:55.019367 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad278afa-71f3-4230-b04b-2e1521a59c28 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:55.021511 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae5376d-1ef2-4ec7-976d-f804fd2c7c58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:55.022698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae5376d-1ef2-4ec7-976d-f804fd2c7c58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:55.022698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae5376d-1ef2-4ec7-976d-f804fd2c7c58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:55.022927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae5376d-1ef2-4ec7-976d-f804fd2c7c58 admin admin] PATH_INFO: `/v3/users/d8a908a871c94799b96874aca602e984` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:55.024143 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae5376d-1ef2-4ec7-976d-f804fd2c7c58 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d8a908a871c94799b96874aca602e984)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:55.026791 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae5376d-1ef2-4ec7-976d-f804fd2c7c58 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:55.036552 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 945/1913] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:54 2022] PUT /identity/v3/projects/0ccb60fdca144211a380fd193c8034b1/users/05fc5bc8a32042608a32cafe4de085bd/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:55.040941 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291e3680-2e68-46dd-8fc8-bb330341dfc3 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:55.162627 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291e3680-2e68-46dd-8fc8-bb330341dfc3 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:55.167950 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8130dc2-df18-4b43-8cea-b0c60b94aa1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:55.168594 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8130dc2-df18-4b43-8cea-b0c60b94aa1b admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:55.168872 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8130dc2-df18-4b43-8cea-b0c60b94aa1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:55.169144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8130dc2-df18-4b43-8cea-b0c60b94aa1b admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:55.169798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8130dc2-df18-4b43-8cea-b0c60b94aa1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:55.172588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8130dc2-df18-4b43-8cea-b0c60b94aa1b admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:55.203600 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 946/1914] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:55 2022] GET /identity/v3/roles => generated 1420 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:55.204927 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8130dc2-df18-4b43-8cea-b0c60b94aa1b admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:55.209177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-b8130dc2-df18-4b43-8cea-b0c60b94aa1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:55.240662 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8130dc2-df18-4b43-8cea-b0c60b94aa1b admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:55.261050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1807ec59-f44c-4d1f-b58c-a98ffcd22503 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:55.264566 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1807ec59-f44c-4d1f-b58c-a98ffcd22503 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:55.264961 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1807ec59-f44c-4d1f-b58c-a98ffcd22503 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:55.265363 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1807ec59-f44c-4d1f-b58c-a98ffcd22503 admin admin] PATH_INFO: `/v3/users/6a17680ec574452a852cfdf9dc647170` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:55.272993 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1807ec59-f44c-4d1f-b58c-a98ffcd22503 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6a17680ec574452a852cfdf9dc647170)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:55.277165 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1807ec59-f44c-4d1f-b58c-a98ffcd22503 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:55.305891 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-2ae5376d-1ef2-4ec7-976d-f804fd2c7c58 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd8a908a871c94799b96874aca602e984'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:55.415229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-2ae5376d-1ef2-4ec7-976d-f804fd2c7c58 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd8a908a871c94799b96874aca602e984'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:55.424806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-2ae5376d-1ef2-4ec7-976d-f804fd2c7c58 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd8a908a871c94799b96874aca602e984'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:55.425662 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-2ae5376d-1ef2-4ec7-976d-f804fd2c7c58 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd8a908a871c94799b96874aca602e984'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:55.434376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-2ae5376d-1ef2-4ec7-976d-f804fd2c7c58 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd8a908a871c94799b96874aca602e984'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:55.441594 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 969/1915] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:54 2022] DELETE /identity/v3/users/d8a908a871c94799b96874aca602e984 => generated 0 bytes in 457 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:55.442441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae5376d-1ef2-4ec7-976d-f804fd2c7c58 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:55.443792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-2ae5376d-1ef2-4ec7-976d-f804fd2c7c58 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:55.508447 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae5376d-1ef2-4ec7-976d-f804fd2c7c58 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:55.513378 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1807ec59-f44c-4d1f-b58c-a98ffcd22503 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6a17680ec574452a852cfdf9dc647170'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:55.517422 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff154f48-bf66-4bcc-836b-776e4a227a5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:55.517871 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff154f48-bf66-4bcc-836b-776e4a227a5b admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:55.518144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff154f48-bf66-4bcc-836b-776e4a227a5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:55.518944 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff154f48-bf66-4bcc-836b-776e4a227a5b admin admin] PATH_INFO: `/v3/projects/0ccb60fdca144211a380fd193c8034b1/users/05fc5bc8a32042608a32cafe4de085bd/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:55.521123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff154f48-bf66-4bcc-836b-776e4a227a5b admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ccb60fdca144211a380fd193c8034b1, user_id=05fc5bc8a32042608a32cafe4de085bd, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:55.523983 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff154f48-bf66-4bcc-836b-776e4a227a5b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:55.549914 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 970/1916] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:55 2022] PUT /identity/v3/projects/0ccb60fdca144211a380fd193c8034b1/users/05fc5bc8a32042608a32cafe4de085bd/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:55.552088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eff076-0045-4da6-a902-7f6f32c2a5cc None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:55.552368 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eff076-0045-4da6-a902-7f6f32c2a5cc None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:55.552662 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6eff076-0045-4da6-a902-7f6f32c2a5cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:55.577142 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-b6eff076-0045-4da6-a902-7f6f32c2a5cc None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:55.578508 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-b6eff076-0045-4da6-a902-7f6f32c2a5cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:55.583460 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1807ec59-f44c-4d1f-b58c-a98ffcd22503 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6a17680ec574452a852cfdf9dc647170'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:55.584426 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1807ec59-f44c-4d1f-b58c-a98ffcd22503 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6a17680ec574452a852cfdf9dc647170'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:55.621197 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1807ec59-f44c-4d1f-b58c-a98ffcd22503 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6a17680ec574452a852cfdf9dc647170'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:55.630537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1807ec59-f44c-4d1f-b58c-a98ffcd22503 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6a17680ec574452a852cfdf9dc647170'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:55.638259 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 947/1917] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:55 2022] DELETE /identity/v3/users/6a17680ec574452a852cfdf9dc647170 => generated 0 bytes in 434 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:55.641504 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1807ec59-f44c-4d1f-b58c-a98ffcd22503 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:55.641781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-1807ec59-f44c-4d1f-b58c-a98ffcd22503 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:55.666612 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 971/1918] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:55.670557 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b4e786-73fc-4609-9c82-6ad7f52f952f None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:55.670557 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b4e786-73fc-4609-9c82-6ad7f52f952f None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:55.670557 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b4e786-73fc-4609-9c82-6ad7f52f952f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:55.704136 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-a2b4e786-73fc-4609-9c82-6ad7f52f952f None None] MFA Rules not processed for user `05fc5bc8a32042608a32cafe4de085bd`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:55.705371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a2b4e786-73fc-4609-9c82-6ad7f52f952f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:55.749848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1807ec59-f44c-4d1f-b58c-a98ffcd22503 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:55.757298 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c085611-b3c7-4389-b6b6-d376024f4b1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:55.757807 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c085611-b3c7-4389-b6b6-d376024f4b1d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:55.760371 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c085611-b3c7-4389-b6b6-d376024f4b1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:55.760371 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c085611-b3c7-4389-b6b6-d376024f4b1d admin admin] PATH_INFO: `/v3/projects/265cc577b49941f88bf16b2475f76f3b` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:55.762008 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c085611-b3c7-4389-b6b6-d376024f4b1d admin admin] RBAC: Authorizing `identity:delete_project(project_id=265cc577b49941f88bf16b2475f76f3b)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:55.765016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c085611-b3c7-4389-b6b6-d376024f4b1d admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:55.802193 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 972/1919] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:55 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:55.803558 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8258cc53-a8b2-4828-94f4-8b21dc8e32da None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:55.803843 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8258cc53-a8b2-4828-94f4-8b21dc8e32da None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:55.804123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8258cc53-a8b2-4828-94f4-8b21dc8e32da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:55.824042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-8258cc53-a8b2-4828-94f4-8b21dc8e32da None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:55.825251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8258cc53-a8b2-4828-94f4-8b21dc8e32da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:55.834013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 973/1920] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:55.834843 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8258cc53-a8b2-4828-94f4-8b21dc8e32da None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:55.838095 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8258cc53-a8b2-4828-94f4-8b21dc8e32da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:55.865095 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:55.868241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5c085611-b3c7-4389-b6b6-d376024f4b1d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '265cc577b49941f88bf16b2475f76f3b', 'request_id': 'req-5c085611-b3c7-4389-b6b6-d376024f4b1d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:55.901501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8258cc53-a8b2-4828-94f4-8b21dc8e32da None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:55.914625 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5c085611-b3c7-4389-b6b6-d376024f4b1d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '265cc577b49941f88bf16b2475f76f3b', 'request_id': 'req-5c085611-b3c7-4389-b6b6-d376024f4b1d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:55.920690 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5c085611-b3c7-4389-b6b6-d376024f4b1d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '265cc577b49941f88bf16b2475f76f3b', 'request_id': 'req-5c085611-b3c7-4389-b6b6-d376024f4b1d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:55.932171 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5c085611-b3c7-4389-b6b6-d376024f4b1d admin admin] The token cache is being invalidate because project 265cc577b49941f88bf16b2475f76f3b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:55.932457 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5c085611-b3c7-4389-b6b6-d376024f4b1d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:55.933128 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5c085611-b3c7-4389-b6b6-d376024f4b1d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:55.935339 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 948/1921] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:55 2022] DELETE /identity/v3/projects/265cc577b49941f88bf16b2475f76f3b => generated 0 bytes in 296 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:55.935453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c085611-b3c7-4389-b6b6-d376024f4b1d admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:55.938525 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-5c085611-b3c7-4389-b6b6-d376024f4b1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:55.953394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21dbac1-7215-42cf-8f81-2b57a56bde38 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:55.953999 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21dbac1-7215-42cf-8f81-2b57a56bde38 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:55.954298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21dbac1-7215-42cf-8f81-2b57a56bde38 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:55.954589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21dbac1-7215-42cf-8f81-2b57a56bde38 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:55.956948 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f21dbac1-7215-42cf-8f81-2b57a56bde38 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:56.001537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c085611-b3c7-4389-b6b6-d376024f4b1d admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:56.007224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f94c069-e94d-4bd9-abd7-702dd2dae6a1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:56.007842 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f94c069-e94d-4bd9-abd7-702dd2dae6a1 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:56.008123 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f94c069-e94d-4bd9-abd7-702dd2dae6a1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:56.008396 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f94c069-e94d-4bd9-abd7-702dd2dae6a1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:56.010129 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-1f94c069-e94d-4bd9-abd7-702dd2dae6a1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:56.023928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:56.037486 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21dbac1-7215-42cf-8f81-2b57a56bde38 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:56.040610 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21dbac1-7215-42cf-8f81-2b57a56bde38 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:56.049279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 974/1922] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:56.050037 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21dbac1-7215-42cf-8f81-2b57a56bde38 service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:56.051568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f21dbac1-7215-42cf-8f81-2b57a56bde38 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:56.067399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21dbac1-7215-42cf-8f81-2b57a56bde38 service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:56.073345 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eccb442-f9ca-49af-a155-ab2f932ead9d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:56.074649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eccb442-f9ca-49af-a155-ab2f932ead9d service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:56.075037 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eccb442-f9ca-49af-a155-ab2f932ead9d service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:56.075729 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eccb442-f9ca-49af-a155-ab2f932ead9d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:56.078936 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-0eccb442-f9ca-49af-a155-ab2f932ead9d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:56.111705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eccb442-f9ca-49af-a155-ab2f932ead9d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:56.115305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eccb442-f9ca-49af-a155-ab2f932ead9d service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:56.137598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 975/1923] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:56.138627 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eccb442-f9ca-49af-a155-ab2f932ead9d service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:56.139933 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f94c069-e94d-4bd9-abd7-702dd2dae6a1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:56.141640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f94c069-e94d-4bd9-abd7-702dd2dae6a1 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:56.142939 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-0eccb442-f9ca-49af-a155-ab2f932ead9d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:56.162912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eccb442-f9ca-49af-a155-ab2f932ead9d service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:56.169616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9999ae0-677b-437a-9858-85ef709e3212 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:56.169880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 949/1924] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:55 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:02:56.172244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9999ae0-677b-437a-9858-85ef709e3212 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:56.172768 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9999ae0-677b-437a-9858-85ef709e3212 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:56.173301 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9999ae0-677b-437a-9858-85ef709e3212 admin admin] PATH_INFO: `/v3/users/d675c12bb79f4c95887529e6b26f00ad` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:56.176078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9999ae0-677b-437a-9858-85ef709e3212 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d675c12bb79f4c95887529e6b26f00ad)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:56.179320 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9999ae0-677b-437a-9858-85ef709e3212 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:56.292012 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f94c069-e94d-4bd9-abd7-702dd2dae6a1 service cinder] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:56.302697 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f94c069-e94d-4bd9-abd7-702dd2dae6a1 service cinder] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:56.306600 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5d498d-9968-48cc-bb35-24c85f9e4258 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:56.307274 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5d498d-9968-48cc-bb35-24c85f9e4258 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:56.307558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5d498d-9968-48cc-bb35-24c85f9e4258 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:56.307826 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5d498d-9968-48cc-bb35-24c85f9e4258 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:56.309869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-1b5d498d-9968-48cc-bb35-24c85f9e4258 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:56.397364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e9999ae0-677b-437a-9858-85ef709e3212 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd675c12bb79f4c95887529e6b26f00ad'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:56.409117 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5d498d-9968-48cc-bb35-24c85f9e4258 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:56.410919 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5d498d-9968-48cc-bb35-24c85f9e4258 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:56.421409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 950/1925] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:02:56 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:02:56.480805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e9999ae0-677b-437a-9858-85ef709e3212 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd675c12bb79f4c95887529e6b26f00ad'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:56.501338 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e9999ae0-677b-437a-9858-85ef709e3212 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd675c12bb79f4c95887529e6b26f00ad'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:56.502052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e9999ae0-677b-437a-9858-85ef709e3212 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd675c12bb79f4c95887529e6b26f00ad'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:56.512396 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e9999ae0-677b-437a-9858-85ef709e3212 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd675c12bb79f4c95887529e6b26f00ad'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:56.525126 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 976/1926] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:56 2022] DELETE /identity/v3/users/d675c12bb79f4c95887529e6b26f00ad => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:56.623718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=85,keystone:INSERT=5,keystone:DELETE=10 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:56.769690 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5d498d-9968-48cc-bb35-24c85f9e4258 service cinder] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:56.773823 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-1b5d498d-9968-48cc-bb35-24c85f9e4258 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:56.800593 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5d498d-9968-48cc-bb35-24c85f9e4258 service cinder] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:56.837509 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9999ae0-677b-437a-9858-85ef709e3212 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:56.839486 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-e9999ae0-677b-437a-9858-85ef709e3212 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:56.968999 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819eb41d-93b7-4413-b6d2-2cf9aad165b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:56.969675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819eb41d-93b7-4413-b6d2-2cf9aad165b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:56.969959 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819eb41d-93b7-4413-b6d2-2cf9aad165b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:56.970377 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819eb41d-93b7-4413-b6d2-2cf9aad165b7 admin admin] PATH_INFO: `/v3/projects/8e7ff127ca7146df872525477b42f11d` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:56.971901 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819eb41d-93b7-4413-b6d2-2cf9aad165b7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e7ff127ca7146df872525477b42f11d)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:56.974410 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819eb41d-93b7-4413-b6d2-2cf9aad165b7 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:57.014098 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9999ae0-677b-437a-9858-85ef709e3212 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:57.019341 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11d6ab7-7950-4561-89ac-6b289bdeddee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:57.020330 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11d6ab7-7950-4561-89ac-6b289bdeddee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:57.020330 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11d6ab7-7950-4561-89ac-6b289bdeddee admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:57.020592 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11d6ab7-7950-4561-89ac-6b289bdeddee admin admin] PATH_INFO: `/v3/projects/00ded4c4066a43b5933e10db7856ff10` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:57.022096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11d6ab7-7950-4561-89ac-6b289bdeddee admin admin] RBAC: Authorizing `identity:delete_project(project_id=00ded4c4066a43b5933e10db7856ff10)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:57.023755 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11d6ab7-7950-4561-89ac-6b289bdeddee admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:57.214950 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-d11d6ab7-7950-4561-89ac-6b289bdeddee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '00ded4c4066a43b5933e10db7856ff10', 'request_id': 'req-d11d6ab7-7950-4561-89ac-6b289bdeddee'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.229722 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-819eb41d-93b7-4413-b6d2-2cf9aad165b7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e7ff127ca7146df872525477b42f11d', 'request_id': 'req-819eb41d-93b7-4413-b6d2-2cf9aad165b7'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.297145 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-d11d6ab7-7950-4561-89ac-6b289bdeddee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '00ded4c4066a43b5933e10db7856ff10', 'request_id': 'req-d11d6ab7-7950-4561-89ac-6b289bdeddee'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.303558 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-d11d6ab7-7950-4561-89ac-6b289bdeddee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '00ded4c4066a43b5933e10db7856ff10', 'request_id': 'req-d11d6ab7-7950-4561-89ac-6b289bdeddee'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.314906 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-d11d6ab7-7950-4561-89ac-6b289bdeddee admin admin] The token cache is being invalidate because project 00ded4c4066a43b5933e10db7856ff10 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:57.316094 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-d11d6ab7-7950-4561-89ac-6b289bdeddee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.316869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-d11d6ab7-7950-4561-89ac-6b289bdeddee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.318369 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 977/1927] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:56 2022] DELETE /identity/v3/projects/00ded4c4066a43b5933e10db7856ff10 => generated 0 bytes in 481 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:57.319252 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11d6ab7-7950-4561-89ac-6b289bdeddee admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:57.320496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d11d6ab7-7950-4561-89ac-6b289bdeddee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:57.382983 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11d6ab7-7950-4561-89ac-6b289bdeddee admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:57.387518 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8148553d-c876-48cf-85bd-8d6d765d27df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:57.388137 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8148553d-c876-48cf-85bd-8d6d765d27df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:57.388413 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8148553d-c876-48cf-85bd-8d6d765d27df admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:57.388680 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8148553d-c876-48cf-85bd-8d6d765d27df admin admin] PATH_INFO: `/v3/projects/2353a6dd9ae24027ae79623008a5d979` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:57.389776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8148553d-c876-48cf-85bd-8d6d765d27df admin admin] RBAC: Authorizing `identity:delete_project(project_id=2353a6dd9ae24027ae79623008a5d979)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:57.391700 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8148553d-c876-48cf-85bd-8d6d765d27df admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:57.445980 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-819eb41d-93b7-4413-b6d2-2cf9aad165b7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e7ff127ca7146df872525477b42f11d', 'request_id': 'req-819eb41d-93b7-4413-b6d2-2cf9aad165b7'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.451908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-819eb41d-93b7-4413-b6d2-2cf9aad165b7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e7ff127ca7146df872525477b42f11d', 'request_id': 'req-819eb41d-93b7-4413-b6d2-2cf9aad165b7'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.475589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-819eb41d-93b7-4413-b6d2-2cf9aad165b7 admin admin] The token cache is being invalidate because project 8e7ff127ca7146df872525477b42f11d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:57.475589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-819eb41d-93b7-4413-b6d2-2cf9aad165b7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.476209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-819eb41d-93b7-4413-b6d2-2cf9aad165b7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.477783 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 951/1928] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:56 2022] DELETE /identity/v3/projects/8e7ff127ca7146df872525477b42f11d => generated 0 bytes in 709 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:57.478814 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819eb41d-93b7-4413-b6d2-2cf9aad165b7 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:57.480249 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-819eb41d-93b7-4413-b6d2-2cf9aad165b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:57.544073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8148553d-c876-48cf-85bd-8d6d765d27df admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2353a6dd9ae24027ae79623008a5d979', 'request_id': 'req-8148553d-c876-48cf-85bd-8d6d765d27df'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.545054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819eb41d-93b7-4413-b6d2-2cf9aad165b7 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:57.601172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f69bd8-fc77-4989-892b-015efb80cbc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:57.602023 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f69bd8-fc77-4989-892b-015efb80cbc9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:57.602139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f69bd8-fc77-4989-892b-015efb80cbc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:57.602350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f69bd8-fc77-4989-892b-015efb80cbc9 admin admin] PATH_INFO: `/v3/users/97e850b435b64bb1ad1837e8ff65e750` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:57.603567 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f69bd8-fc77-4989-892b-015efb80cbc9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=97e850b435b64bb1ad1837e8ff65e750)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:57.605531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f69bd8-fc77-4989-892b-015efb80cbc9 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:57.645719 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8148553d-c876-48cf-85bd-8d6d765d27df admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2353a6dd9ae24027ae79623008a5d979', 'request_id': 'req-8148553d-c876-48cf-85bd-8d6d765d27df'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.651908 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8148553d-c876-48cf-85bd-8d6d765d27df admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2353a6dd9ae24027ae79623008a5d979', 'request_id': 'req-8148553d-c876-48cf-85bd-8d6d765d27df'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.663107 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8148553d-c876-48cf-85bd-8d6d765d27df admin admin] The token cache is being invalidate because project 2353a6dd9ae24027ae79623008a5d979 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:57.663422 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8148553d-c876-48cf-85bd-8d6d765d27df admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.664065 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8148553d-c876-48cf-85bd-8d6d765d27df admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.665555 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 978/1929] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:57 2022] DELETE /identity/v3/projects/2353a6dd9ae24027ae79623008a5d979 => generated 0 bytes in 347 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:57.666313 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8148553d-c876-48cf-85bd-8d6d765d27df admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:57.667621 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8148553d-c876-48cf-85bd-8d6d765d27df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:57.681045 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8148553d-c876-48cf-85bd-8d6d765d27df admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:57.685990 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63ad690-8fcc-4b20-9317-94cbb24ecba5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:57.686788 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63ad690-8fcc-4b20-9317-94cbb24ecba5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:57.686946 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63ad690-8fcc-4b20-9317-94cbb24ecba5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:57.687209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63ad690-8fcc-4b20-9317-94cbb24ecba5 admin admin] PATH_INFO: `/v3/users/875446d2354e44a18b8db47188ac9b48` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:57.688312 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63ad690-8fcc-4b20-9317-94cbb24ecba5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=875446d2354e44a18b8db47188ac9b48)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:57.690982 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63ad690-8fcc-4b20-9317-94cbb24ecba5 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:57.722698 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:57.800094 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-b7f69bd8-fc77-4989-892b-015efb80cbc9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '97e850b435b64bb1ad1837e8ff65e750'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.869354 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-b7f69bd8-fc77-4989-892b-015efb80cbc9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '97e850b435b64bb1ad1837e8ff65e750'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.870089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-b7f69bd8-fc77-4989-892b-015efb80cbc9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '97e850b435b64bb1ad1837e8ff65e750'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.884580 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-b7f69bd8-fc77-4989-892b-015efb80cbc9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '97e850b435b64bb1ad1837e8ff65e750'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.888458 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e63ad690-8fcc-4b20-9317-94cbb24ecba5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '875446d2354e44a18b8db47188ac9b48'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.892109 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-b7f69bd8-fc77-4989-892b-015efb80cbc9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '97e850b435b64bb1ad1837e8ff65e750'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:57.898281 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 952/1930] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:57 2022] DELETE /identity/v3/users/97e850b435b64bb1ad1837e8ff65e750 => generated 0 bytes in 420 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:57.899605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ba9240-be26-44e8-87a5-20e8036dffff None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:57.899890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ba9240-be26-44e8-87a5-20e8036dffff None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:57.900157 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ba9240-be26-44e8-87a5-20e8036dffff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:57.920491 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-66ba9240-be26-44e8-87a5-20e8036dffff None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:57.921769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-66ba9240-be26-44e8-87a5-20e8036dffff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:57.978597 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e63ad690-8fcc-4b20-9317-94cbb24ecba5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '875446d2354e44a18b8db47188ac9b48'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:58.000469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e63ad690-8fcc-4b20-9317-94cbb24ecba5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '875446d2354e44a18b8db47188ac9b48'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:58.001276 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e63ad690-8fcc-4b20-9317-94cbb24ecba5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '875446d2354e44a18b8db47188ac9b48'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:58.014487 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 953/1931] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:58.016029 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e63ad690-8fcc-4b20-9317-94cbb24ecba5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '875446d2354e44a18b8db47188ac9b48'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:58.017172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb66dda-00ce-4746-a95b-5a44d0f1296e None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:58.017590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb66dda-00ce-4746-a95b-5a44d0f1296e None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:58.018040 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb66dda-00ce-4746-a95b-5a44d0f1296e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:58.025840 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 979/1932] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:02:57 2022] DELETE /identity/v3/users/875446d2354e44a18b8db47188ac9b48 => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:58.054934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-bcb66dda-00ce-4746-a95b-5a44d0f1296e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:58.055292 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-bcb66dda-00ce-4746-a95b-5a44d0f1296e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:58.151660 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 954/1933] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:02:58.155798 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63ad690-8fcc-4b20-9317-94cbb24ecba5 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:58.213430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:58.218070 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63ad690-8fcc-4b20-9317-94cbb24ecba5 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:58.480844 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b70c4fd-1d0c-4126-96bb-ba398db4331a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:58.480844 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b70c4fd-1d0c-4126-96bb-ba398db4331a admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:58.480844 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b70c4fd-1d0c-4126-96bb-ba398db4331a admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:58.480844 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b70c4fd-1d0c-4126-96bb-ba398db4331a admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:58.487786 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b70c4fd-1d0c-4126-96bb-ba398db4331a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:58.487786 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b70c4fd-1d0c-4126-96bb-ba398db4331a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:58.487786 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b70c4fd-1d0c-4126-96bb-ba398db4331a admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:58.487786 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 980/1934] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:02:58.489168 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb66dda-00ce-4746-a95b-5a44d0f1296e None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:58.489168 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb66dda-00ce-4746-a95b-5a44d0f1296e None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:58.489168 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d2736a-36b6-44c1-9f53-15a6d754b647 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:58.489168 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d2736a-36b6-44c1-9f53-15a6d754b647 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:58.493495 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d2736a-36b6-44c1-9f53-15a6d754b647 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:58.493495 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d2736a-36b6-44c1-9f53-15a6d754b647 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:58.493495 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d2736a-36b6-44c1-9f53-15a6d754b647 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:58.493495 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d2736a-36b6-44c1-9f53-15a6d754b647 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:58.493495 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 955/1935] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:58 2022] POST /identity/v3/projects => generated 385 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:58.495181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b70c4fd-1d0c-4126-96bb-ba398db4331a admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:58.506089 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b70c4fd-1d0c-4126-96bb-ba398db4331a admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:58.579623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d2736a-36b6-44c1-9f53-15a6d754b647 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:58.582371 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-44d2736a-36b6-44c1-9f53-15a6d754b647 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:58.606866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5b3bc4-cf79-4ab4-b553-65def6cd5640 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:58.607489 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5b3bc4-cf79-4ab4-b553-65def6cd5640 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:58.608108 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5b3bc4-cf79-4ab4-b553-65def6cd5640 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:58.608537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5b3bc4-cf79-4ab4-b553-65def6cd5640 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:58.609331 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e5b3bc4-cf79-4ab4-b553-65def6cd5640 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:58.612791 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e5b3bc4-cf79-4ab4-b553-65def6cd5640 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:58.661843 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d2736a-36b6-44c1-9f53-15a6d754b647 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:58.667379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6913b9-7265-4642-ab25-12f64d180919 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:58.668018 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6913b9-7265-4642-ab25-12f64d180919 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:58.668292 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6913b9-7265-4642-ab25-12f64d180919 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:58.668560 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6913b9-7265-4642-ab25-12f64d180919 admin admin] PATH_INFO: `/v3/projects/75a7e741fe094d57a6c751714e82f6e9` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:58.669725 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6913b9-7265-4642-ab25-12f64d180919 admin admin] RBAC: Authorizing `identity:delete_project(project_id=75a7e741fe094d57a6c751714e82f6e9)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:58.671829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6913b9-7265-4642-ab25-12f64d180919 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:58.715114 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 981/1936] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:58 2022] POST /identity/v3/users => generated 293 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:58.716269 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5b3bc4-cf79-4ab4-b553-65def6cd5640 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:58.718021 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-2e5b3bc4-cf79-4ab4-b553-65def6cd5640 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:58.731028 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5b3bc4-cf79-4ab4-b553-65def6cd5640 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:58.735943 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789342e1-3e47-41be-8d7c-1c90617b7bdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:58.736563 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789342e1-3e47-41be-8d7c-1c90617b7bdb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:58.736876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789342e1-3e47-41be-8d7c-1c90617b7bdb admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:58.737149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789342e1-3e47-41be-8d7c-1c90617b7bdb admin admin] PATH_INFO: `/v3/projects/50b01a27686b416f90747fa4f4f80a9b` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:58.738346 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789342e1-3e47-41be-8d7c-1c90617b7bdb admin admin] RBAC: Authorizing `identity:delete_project(project_id=50b01a27686b416f90747fa4f4f80a9b)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:58.740211 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789342e1-3e47-41be-8d7c-1c90617b7bdb admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:58.820872 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-0e6913b9-7265-4642-ab25-12f64d180919 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '75a7e741fe094d57a6c751714e82f6e9', 'request_id': 'req-0e6913b9-7265-4642-ab25-12f64d180919'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:58.869593 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-789342e1-3e47-41be-8d7c-1c90617b7bdb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '50b01a27686b416f90747fa4f4f80a9b', 'request_id': 'req-789342e1-3e47-41be-8d7c-1c90617b7bdb'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:58.955561 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-0e6913b9-7265-4642-ab25-12f64d180919 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '75a7e741fe094d57a6c751714e82f6e9', 'request_id': 'req-0e6913b9-7265-4642-ab25-12f64d180919'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:58.964774 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-0e6913b9-7265-4642-ab25-12f64d180919 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '75a7e741fe094d57a6c751714e82f6e9', 'request_id': 'req-0e6913b9-7265-4642-ab25-12f64d180919'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:58.988023 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-0e6913b9-7265-4642-ab25-12f64d180919 admin admin] The token cache is being invalidate because project 75a7e741fe094d57a6c751714e82f6e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:58.988247 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-0e6913b9-7265-4642-ab25-12f64d180919 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:58.989075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-0e6913b9-7265-4642-ab25-12f64d180919 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:58.990926 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 956/1937] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:58 2022] DELETE /identity/v3/projects/75a7e741fe094d57a6c751714e82f6e9 => generated 0 bytes in 412 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:02:58.998976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6913b9-7265-4642-ab25-12f64d180919 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:58.998976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-0e6913b9-7265-4642-ab25-12f64d180919 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:59.020833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-789342e1-3e47-41be-8d7c-1c90617b7bdb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '50b01a27686b416f90747fa4f4f80a9b', 'request_id': 'req-789342e1-3e47-41be-8d7c-1c90617b7bdb'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:59.028286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-789342e1-3e47-41be-8d7c-1c90617b7bdb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '50b01a27686b416f90747fa4f4f80a9b', 'request_id': 'req-789342e1-3e47-41be-8d7c-1c90617b7bdb'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:59.040919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-789342e1-3e47-41be-8d7c-1c90617b7bdb admin admin] The token cache is being invalidate because project 50b01a27686b416f90747fa4f4f80a9b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:02:59.041477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-789342e1-3e47-41be-8d7c-1c90617b7bdb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:59.042137 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-789342e1-3e47-41be-8d7c-1c90617b7bdb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:02:59.043682 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 982/1938] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:02:58 2022] DELETE /identity/v3/projects/50b01a27686b416f90747fa4f4f80a9b => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:02:59.044870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14415223-9c8e-4899-bada-06239368d11f None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:59.045148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14415223-9c8e-4899-bada-06239368d11f None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:59.045414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14415223-9c8e-4899-bada-06239368d11f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:59.077267 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-14415223-9c8e-4899-bada-06239368d11f None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:59.078590 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-14415223-9c8e-4899-bada-06239368d11f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:59.078769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6913b9-7265-4642-ab25-12f64d180919 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:59.135555 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3f1fc2-8542-4a43-b264-700cef516024 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:59.138783 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3f1fc2-8542-4a43-b264-700cef516024 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:59.139127 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3f1fc2-8542-4a43-b264-700cef516024 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:59.139410 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3f1fc2-8542-4a43-b264-700cef516024 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:59.140073 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da3f1fc2-8542-4a43-b264-700cef516024 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:59.141690 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da3f1fc2-8542-4a43-b264-700cef516024 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:59.145034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 983/1939] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:59.146507 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f095c31-3eb9-4583-a667-dbfdd5db29bd None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:59.150954 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f095c31-3eb9-4583-a667-dbfdd5db29bd None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:59.150954 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f095c31-3eb9-4583-a667-dbfdd5db29bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:59.156177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 957/1940] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:58 2022] GET /identity/v3/roles => generated 1420 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:59.156177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdf33c1-64ad-403e-b636-311a86abdfa3 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:59.156177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdf33c1-64ad-403e-b636-311a86abdfa3 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:59.156177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdf33c1-64ad-403e-b636-311a86abdfa3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:59.172238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-6f095c31-3eb9-4583-a667-dbfdd5db29bd None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:59.173516 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-6f095c31-3eb9-4583-a667-dbfdd5db29bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:59.177480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-4fdf33c1-64ad-403e-b636-311a86abdfa3 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:59.178727 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-4fdf33c1-64ad-403e-b636-311a86abdfa3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:59.182606 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 984/1941] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:59.185523 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f095c31-3eb9-4583-a667-dbfdd5db29bd None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:59.185523 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-6f095c31-3eb9-4583-a667-dbfdd5db29bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:59.186961 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 958/1942] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:59.191172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdf33c1-64ad-403e-b636-311a86abdfa3 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:59.197106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f095c31-3eb9-4583-a667-dbfdd5db29bd None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:59.200823 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdf33c1-64ad-403e-b636-311a86abdfa3 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:59.205406 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2e0a0c-53de-43f2-95fb-c9cf59169ce7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:59.206236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2331ab15-fec1-423d-8515-4c5bb811ddbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:59.206751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2e0a0c-53de-43f2-95fb-c9cf59169ce7 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:59.207095 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2331ab15-fec1-423d-8515-4c5bb811ddbd admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:59.207206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2e0a0c-53de-43f2-95fb-c9cf59169ce7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:59.207394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2331ab15-fec1-423d-8515-4c5bb811ddbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:59.207489 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2e0a0c-53de-43f2-95fb-c9cf59169ce7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:59.207642 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2331ab15-fec1-423d-8515-4c5bb811ddbd admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:59.208014 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2e0a0c-53de-43f2-95fb-c9cf59169ce7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:59.208315 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2e0a0c-53de-43f2-95fb-c9cf59169ce7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:59.208435 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2331ab15-fec1-423d-8515-4c5bb811ddbd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:59.209897 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2e0a0c-53de-43f2-95fb-c9cf59169ce7 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:59.210783 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2331ab15-fec1-423d-8515-4c5bb811ddbd admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:59.220734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:59.225520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 985/1943] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:59 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:59.225520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedcba45-c526-49e2-9400-c470577230fe None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:59.225520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedcba45-c526-49e2-9400-c470577230fe None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:59.225520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedcba45-c526-49e2-9400-c470577230fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:59.225926 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 959/1944] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 14:02:59.227490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2e0a0c-53de-43f2-95fb-c9cf59169ce7 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:59.237365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2e0a0c-53de-43f2-95fb-c9cf59169ce7 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:59.245736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b6507c-bd5a-471c-aa29-30482bd97b81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:59.245736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b6507c-bd5a-471c-aa29-30482bd97b81 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:59.245736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b6507c-bd5a-471c-aa29-30482bd97b81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:59.245736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b6507c-bd5a-471c-aa29-30482bd97b81 admin admin] PATH_INFO: `/v3/projects/02c99699ea244fee9ad03b9f12c512dc/users/8e60004b6011499fb5a914bedc7b47a3/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:59.246141 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-aedcba45-c526-49e2-9400-c470577230fe None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:59.246141 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-aedcba45-c526-49e2-9400-c470577230fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:59.253876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 986/1945] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:59.254695 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedcba45-c526-49e2-9400-c470577230fe None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:59.259468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b6507c-bd5a-471c-aa29-30482bd97b81 admin admin] RBAC: Authorizing `identity:create_grant(project_id=02c99699ea244fee9ad03b9f12c512dc, user_id=8e60004b6011499fb5a914bedc7b47a3, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:59.262524 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28b6507c-bd5a-471c-aa29-30482bd97b81 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:59.264233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedcba45-c526-49e2-9400-c470577230fe None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:59.269810 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e28b6dcb-7796-4f4c-bb9a-468931fd28b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:59.270715 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28b6dcb-7796-4f4c-bb9a-468931fd28b8 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:59.270994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28b6dcb-7796-4f4c-bb9a-468931fd28b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:59.271279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e28b6dcb-7796-4f4c-bb9a-468931fd28b8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:59.271890 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28b6dcb-7796-4f4c-bb9a-468931fd28b8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:59.274147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e28b6dcb-7796-4f4c-bb9a-468931fd28b8 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:59.372381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 960/1946] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:59 2022] PUT /identity/v3/projects/02c99699ea244fee9ad03b9f12c512dc/users/8e60004b6011499fb5a914bedc7b47a3/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:02:59.379632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b6507c-bd5a-471c-aa29-30482bd97b81 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:59.410306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 987/1947] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:59 2022] POST /identity/v3/projects => generated 391 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:59.415121 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551b824a-0651-43e3-b1f3-380856a77c28 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:59.415503 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551b824a-0651-43e3-b1f3-380856a77c28 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:59.415874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551b824a-0651-43e3-b1f3-380856a77c28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:59.455310 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-551b824a-0651-43e3-b1f3-380856a77c28 None None] MFA Rules not processed for user `8e60004b6011499fb5a914bedc7b47a3`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:02:59.474363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-551b824a-0651-43e3-b1f3-380856a77c28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:02:59.477242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28b6507c-bd5a-471c-aa29-30482bd97b81 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:59.551190 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 988/1948] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:02:59 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:02:59.552500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551b824a-0651-43e3-b1f3-380856a77c28 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:59.557154 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17d6223-0977-4988-b9b9-9f2e98f0398d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:59.557791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17d6223-0977-4988-b9b9-9f2e98f0398d admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:59.558071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17d6223-0977-4988-b9b9-9f2e98f0398d admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:59.558369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17d6223-0977-4988-b9b9-9f2e98f0398d admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:59.559024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17d6223-0977-4988-b9b9-9f2e98f0398d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:02:59.559330 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17d6223-0977-4988-b9b9-9f2e98f0398d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:59.560926 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17d6223-0977-4988-b9b9-9f2e98f0398d admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:59.567940 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551b824a-0651-43e3-b1f3-380856a77c28 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:59.573664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1c0820-97f5-44d3-9c46-8be05d3aa641 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:59.574482 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1c0820-97f5-44d3-9c46-8be05d3aa641 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:59.578199 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1c0820-97f5-44d3-9c46-8be05d3aa641 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:59.578320 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 961/1949] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:02:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:02:59.578616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1c0820-97f5-44d3-9c46-8be05d3aa641 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:59.578616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1c0820-97f5-44d3-9c46-8be05d3aa641 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:59.578806 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17d6223-0977-4988-b9b9-9f2e98f0398d admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:59.579193 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1c0820-97f5-44d3-9c46-8be05d3aa641 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:59.598473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17d6223-0977-4988-b9b9-9f2e98f0398d admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:59.603310 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba448f9-8c77-4f3b-9699-e735dbd5ab17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:59.603869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba448f9-8c77-4f3b-9699-e735dbd5ab17 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:59.604152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba448f9-8c77-4f3b-9699-e735dbd5ab17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:59.604423 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba448f9-8c77-4f3b-9699-e735dbd5ab17 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:59.605097 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba448f9-8c77-4f3b-9699-e735dbd5ab17 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:59.607221 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba448f9-8c77-4f3b-9699-e735dbd5ab17 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:59.683609 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 989/1950] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:59 2022] POST /identity/v3/users => generated 296 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:59.683609 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1c0820-97f5-44d3-9c46-8be05d3aa641 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:59.696776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1c0820-97f5-44d3-9c46-8be05d3aa641 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:59.701846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36b3337-a536-4462-81fc-3f509c303fb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:59.702490 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36b3337-a536-4462-81fc-3f509c303fb6 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:59.702767 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36b3337-a536-4462-81fc-3f509c303fb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:59.703033 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36b3337-a536-4462-81fc-3f509c303fb6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:59.703663 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36b3337-a536-4462-81fc-3f509c303fb6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:59.705604 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b36b3337-a536-4462-81fc-3f509c303fb6 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:59.719347 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 962/1951] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:59 2022] POST /identity/v3/projects => generated 385 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:59.720227 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba448f9-8c77-4f3b-9699-e735dbd5ab17 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:59.786242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba448f9-8c77-4f3b-9699-e735dbd5ab17 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:59.791175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacd2a4d-1aea-4a9a-83dd-09582231c779 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:59.791781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacd2a4d-1aea-4a9a-83dd-09582231c779 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:59.792065 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacd2a4d-1aea-4a9a-83dd-09582231c779 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:59.792336 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dacd2a4d-1aea-4a9a-83dd-09582231c779 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:59.793183 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dacd2a4d-1aea-4a9a-83dd-09582231c779 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:59.794693 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dacd2a4d-1aea-4a9a-83dd-09582231c779 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:59.826354 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 963/1952] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:59 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:59.826354 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacd2a4d-1aea-4a9a-83dd-09582231c779 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:59.834210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 990/1953] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:02:59 2022] POST /identity/v3/projects => generated 387 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:59.834210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36b3337-a536-4462-81fc-3f509c303fb6 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:59.838876 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dacd2a4d-1aea-4a9a-83dd-09582231c779 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:59.911125 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe9604c-57ca-4e3f-9565-a68274474b03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:59.911954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe9604c-57ca-4e3f-9565-a68274474b03 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:59.912411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe9604c-57ca-4e3f-9565-a68274474b03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:59.912981 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe9604c-57ca-4e3f-9565-a68274474b03 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:59.913969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe9604c-57ca-4e3f-9565-a68274474b03 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:59.916038 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36b3337-a536-4462-81fc-3f509c303fb6 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:59.916186 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe9604c-57ca-4e3f-9565-a68274474b03 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:59.922263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65b4e46-0e3e-4cf4-9700-2445141ce2a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:59.926944 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65b4e46-0e3e-4cf4-9700-2445141ce2a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:59.926944 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65b4e46-0e3e-4cf4-9700-2445141ce2a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:59.926944 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65b4e46-0e3e-4cf4-9700-2445141ce2a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:59.926944 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65b4e46-0e3e-4cf4-9700-2445141ce2a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:59.927262 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65b4e46-0e3e-4cf4-9700-2445141ce2a4 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:59.937814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 991/1954] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:02:59 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:02:59.938629 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65b4e46-0e3e-4cf4-9700-2445141ce2a4 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:59.949233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65b4e46-0e3e-4cf4-9700-2445141ce2a4 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:59.954224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d60cbe-06c2-4943-a6f6-57efbf2211c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:02:59.954852 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d60cbe-06c2-4943-a6f6-57efbf2211c1 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:02:59.955136 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d60cbe-06c2-4943-a6f6-57efbf2211c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:02:59.956125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d60cbe-06c2-4943-a6f6-57efbf2211c1 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:02:59.957351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d60cbe-06c2-4943-a6f6-57efbf2211c1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:02:59.959391 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d60cbe-06c2-4943-a6f6-57efbf2211c1 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:02:59.985959 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 964/1955] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:59 2022] POST /identity/v3/users => generated 299 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:02:59.985959 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe9604c-57ca-4e3f-9565-a68274474b03 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:02:59.994573 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:DELETE=3,keystone:SELECT=97 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:02:59.994777 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe9604c-57ca-4e3f-9565-a68274474b03 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:02:59.999585 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f149e147-fbd8-4571-929f-59829d4bbb35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:00.001371 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f149e147-fbd8-4571-929f-59829d4bbb35 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:00.001371 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f149e147-fbd8-4571-929f-59829d4bbb35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:00.001371 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f149e147-fbd8-4571-929f-59829d4bbb35 admin admin] PATH_INFO: `/v3/projects/ad95901102c9425583cf14ee0698d656/users/fe0780d583af4579954b393f0798cce7/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:00.019392 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f149e147-fbd8-4571-929f-59829d4bbb35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ad95901102c9425583cf14ee0698d656, user_id=fe0780d583af4579954b393f0798cce7, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:00.022164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f149e147-fbd8-4571-929f-59829d4bbb35 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:00.049264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 992/1956] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:02:59 2022] POST /identity/v3/users => generated 294 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:00.049264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d60cbe-06c2-4943-a6f6-57efbf2211c1 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:00.057223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d60cbe-06c2-4943-a6f6-57efbf2211c1 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:00.062013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69ce272-45e3-4432-9547-c06aab829784 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:00.066705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69ce272-45e3-4432-9547-c06aab829784 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:00.066705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69ce272-45e3-4432-9547-c06aab829784 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:00.066705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69ce272-45e3-4432-9547-c06aab829784 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:00.066705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69ce272-45e3-4432-9547-c06aab829784 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:00.066705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69ce272-45e3-4432-9547-c06aab829784 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:00.078233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 993/1957] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:00 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:00.079102 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69ce272-45e3-4432-9547-c06aab829784 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:00.091013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69ce272-45e3-4432-9547-c06aab829784 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:00.095494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d799aa7-9c1e-4bb7-85e9-4b3e920c0cb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:00.096202 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d799aa7-9c1e-4bb7-85e9-4b3e920c0cb5 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:00.096984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d799aa7-9c1e-4bb7-85e9-4b3e920c0cb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:00.097252 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d799aa7-9c1e-4bb7-85e9-4b3e920c0cb5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:00.097899 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d799aa7-9c1e-4bb7-85e9-4b3e920c0cb5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:00.099512 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d799aa7-9c1e-4bb7-85e9-4b3e920c0cb5 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:00.110115 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 994/1958] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:00 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:00.114007 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d799aa7-9c1e-4bb7-85e9-4b3e920c0cb5 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:00.118100 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 965/1959] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:02:59 2022] PUT /identity/v3/projects/ad95901102c9425583cf14ee0698d656/users/fe0780d583af4579954b393f0798cce7/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:00.126397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f149e147-fbd8-4571-929f-59829d4bbb35 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:00.132603 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d799aa7-9c1e-4bb7-85e9-4b3e920c0cb5 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:00.201638 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f149e147-fbd8-4571-929f-59829d4bbb35 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:00.210104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f18d8a-94ef-4bb9-ac91-d1aee4d3af7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:00.210918 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f18d8a-94ef-4bb9-ac91-d1aee4d3af7a admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:00.211206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f18d8a-94ef-4bb9-ac91-d1aee4d3af7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:00.211474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f18d8a-94ef-4bb9-ac91-d1aee4d3af7a admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:00.212225 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f18d8a-94ef-4bb9-ac91-d1aee4d3af7a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:00.214918 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f18d8a-94ef-4bb9-ac91-d1aee4d3af7a admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:00.227760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 966/1960] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:00 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:00.227760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a0d617-28cc-4434-81f3-b61ac9e9afbc None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:00.227760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a0d617-28cc-4434-81f3-b61ac9e9afbc None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:00.227760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a0d617-28cc-4434-81f3-b61ac9e9afbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:00.240318 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e62e408-65a6-4024-885f-98355cf3d116 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:00.241333 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e62e408-65a6-4024-885f-98355cf3d116 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:00.241705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e62e408-65a6-4024-885f-98355cf3d116 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:00.242096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e62e408-65a6-4024-885f-98355cf3d116 admin admin] PATH_INFO: `/v3/projects/62db6dcb146444f997929364e3ba7fd7/users/8152d89a5a634b698869692856bbbbb6/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:00.263323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e62e408-65a6-4024-885f-98355cf3d116 admin admin] RBAC: Authorizing `identity:create_grant(project_id=62db6dcb146444f997929364e3ba7fd7, user_id=8152d89a5a634b698869692856bbbbb6, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:00.266107 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e62e408-65a6-4024-885f-98355cf3d116 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:00.275309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-b9a0d617-28cc-4434-81f3-b61ac9e9afbc None None] MFA Rules not processed for user `fe0780d583af4579954b393f0798cce7`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:00.276658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-b9a0d617-28cc-4434-81f3-b61ac9e9afbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:00.374271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 967/1961] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:00 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:00.375343 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a0d617-28cc-4434-81f3-b61ac9e9afbc None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:00.386701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a0d617-28cc-4434-81f3-b61ac9e9afbc None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:00.390938 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 995/1962] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:00 2022] PUT /identity/v3/projects/62db6dcb146444f997929364e3ba7fd7/users/8152d89a5a634b698869692856bbbbb6/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:00.390938 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e62e408-65a6-4024-885f-98355cf3d116 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:00.467284 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a37e4c0-61ab-4b65-b008-8c8c6902bf17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:00.467417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a37e4c0-61ab-4b65-b008-8c8c6902bf17 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:00.467691 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a37e4c0-61ab-4b65-b008-8c8c6902bf17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:00.467874 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a37e4c0-61ab-4b65-b008-8c8c6902bf17 admin admin] PATH_INFO: `/v3/projects/537ad5add80d45e59823a73b177657d1/users/153768c660764af6af76fe7a6b23a68a/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:00.476081 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e62e408-65a6-4024-885f-98355cf3d116 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:00.483137 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe75909-a167-4da9-9bc6-eaf61b9549b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:00.483755 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe75909-a167-4da9-9bc6-eaf61b9549b3 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:00.484128 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe75909-a167-4da9-9bc6-eaf61b9549b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:00.484378 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe75909-a167-4da9-9bc6-eaf61b9549b3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:00.485315 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe75909-a167-4da9-9bc6-eaf61b9549b3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:00.488730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe75909-a167-4da9-9bc6-eaf61b9549b3 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:00.491966 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a37e4c0-61ab-4b65-b008-8c8c6902bf17 admin admin] RBAC: Authorizing `identity:create_grant(project_id=537ad5add80d45e59823a73b177657d1, user_id=153768c660764af6af76fe7a6b23a68a, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:00.492455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a37e4c0-61ab-4b65-b008-8c8c6902bf17 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:00.606534 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 968/1963] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:00 2022] PUT /identity/v3/projects/537ad5add80d45e59823a73b177657d1/users/153768c660764af6af76fe7a6b23a68a/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:00.607339 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 996/1964] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:00 2022] POST /identity/v3/projects => generated 391 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:00.607430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a37e4c0-61ab-4b65-b008-8c8c6902bf17 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:00.607513 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe75909-a167-4da9-9bc6-eaf61b9549b3 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:00.671254 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a37e4c0-61ab-4b65-b008-8c8c6902bf17 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:00.676256 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca82ef5c-23d2-4756-80f0-c3c65e9b4c9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:00.676836 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe75909-a167-4da9-9bc6-eaf61b9549b3 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:00.677118 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca82ef5c-23d2-4756-80f0-c3c65e9b4c9d admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:00.677396 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca82ef5c-23d2-4756-80f0-c3c65e9b4c9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:00.677670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca82ef5c-23d2-4756-80f0-c3c65e9b4c9d admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:00.678314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca82ef5c-23d2-4756-80f0-c3c65e9b4c9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:00.679957 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca82ef5c-23d2-4756-80f0-c3c65e9b4c9d admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:00.683505 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6deccf4-eeb8-44ce-83ef-fb663d9465d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:00.684114 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6deccf4-eeb8-44ce-83ef-fb663d9465d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:00.684694 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6deccf4-eeb8-44ce-83ef-fb663d9465d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:00.684694 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6deccf4-eeb8-44ce-83ef-fb663d9465d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:00.685272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6deccf4-eeb8-44ce-83ef-fb663d9465d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:00.687249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6deccf4-eeb8-44ce-83ef-fb663d9465d4 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:00.697126 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 969/1965] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:00 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:00.697126 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d45658-96b0-4e3a-9040-6d9a9cbe68ba None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:00.697126 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d45658-96b0-4e3a-9040-6d9a9cbe68ba None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:00.697126 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d45658-96b0-4e3a-9040-6d9a9cbe68ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:00.743430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-00d45658-96b0-4e3a-9040-6d9a9cbe68ba None None] MFA Rules not processed for user `153768c660764af6af76fe7a6b23a68a`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:00.745421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-00d45658-96b0-4e3a-9040-6d9a9cbe68ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:00.783439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:00.790929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 997/1966] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:00 2022] POST /identity/v3/users => generated 302 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:00.792200 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6deccf4-eeb8-44ce-83ef-fb663d9465d4 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:00.811347 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6deccf4-eeb8-44ce-83ef-fb663d9465d4 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:00.826950 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419aa8ce-d8c2-4a08-9deb-2970c4d19814 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:00.827633 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419aa8ce-d8c2-4a08-9deb-2970c4d19814 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:00.827919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419aa8ce-d8c2-4a08-9deb-2970c4d19814 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:00.832396 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419aa8ce-d8c2-4a08-9deb-2970c4d19814 admin admin] PATH_INFO: `/v3/projects/62db6dcb146444f997929364e3ba7fd7/users/8152d89a5a634b698869692856bbbbb6/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:00.834857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419aa8ce-d8c2-4a08-9deb-2970c4d19814 admin admin] RBAC: Authorizing `identity:create_grant(project_id=62db6dcb146444f997929364e3ba7fd7, user_id=8152d89a5a634b698869692856bbbbb6, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:00.843812 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419aa8ce-d8c2-4a08-9deb-2970c4d19814 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:00.853283 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 970/1967] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:00 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:00.854294 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d45658-96b0-4e3a-9040-6d9a9cbe68ba None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:00.871467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d45658-96b0-4e3a-9040-6d9a9cbe68ba None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:00.873579 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 998/1968] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:00 2022] PUT /identity/v3/projects/62db6dcb146444f997929364e3ba7fd7/users/8152d89a5a634b698869692856bbbbb6/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:00.875568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419aa8ce-d8c2-4a08-9deb-2970c4d19814 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:00.932851 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:00.968501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419aa8ce-d8c2-4a08-9deb-2970c4d19814 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:00.974812 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e757d45c-6a2d-4447-a0fb-3e1f7ca1fb2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:00.975362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e757d45c-6a2d-4447-a0fb-3e1f7ca1fb2b admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:00.975971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e757d45c-6a2d-4447-a0fb-3e1f7ca1fb2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:00.976078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e757d45c-6a2d-4447-a0fb-3e1f7ca1fb2b admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:00.976742 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e757d45c-6a2d-4447-a0fb-3e1f7ca1fb2b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:00.979455 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e757d45c-6a2d-4447-a0fb-3e1f7ca1fb2b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:00.998251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b949e51-a56c-49ef-a0d8-27a207ba92cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:00.998945 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b949e51-a56c-49ef-a0d8-27a207ba92cd admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:00.999682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b949e51-a56c-49ef-a0d8-27a207ba92cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:01.000063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b949e51-a56c-49ef-a0d8-27a207ba92cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:01.000873 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b949e51-a56c-49ef-a0d8-27a207ba92cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:01.003143 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b949e51-a56c-49ef-a0d8-27a207ba92cd admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:01.021706 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 971/1969] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:00 2022] GET /identity/v3/roles => generated 1420 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:01.023706 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64840ca0-c92d-4f38-80f0-6eff6e34225f None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:01.024184 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64840ca0-c92d-4f38-80f0-6eff6e34225f None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:01.024733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64840ca0-c92d-4f38-80f0-6eff6e34225f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:01.059736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-64840ca0-c92d-4f38-80f0-6eff6e34225f None None] MFA Rules not processed for user `8152d89a5a634b698869692856bbbbb6`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:01.061140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-64840ca0-c92d-4f38-80f0-6eff6e34225f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:01.078177 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 999/1970] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:00 2022] POST /identity/v3/projects => generated 387 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:01.079168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e757d45c-6a2d-4447-a0fb-3e1f7ca1fb2b admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:01.157815 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e757d45c-6a2d-4447-a0fb-3e1f7ca1fb2b admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:01.162991 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290cc40b-2bd3-414b-ad0b-04a5e97e1941 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:01.164436 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290cc40b-2bd3-414b-ad0b-04a5e97e1941 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:01.165093 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290cc40b-2bd3-414b-ad0b-04a5e97e1941 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:01.165549 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290cc40b-2bd3-414b-ad0b-04a5e97e1941 admin admin] PATH_INFO: `/v3/projects/403ed8c254684a8e87ee70b92a2dbf11/users/029d1350d2ec488c8a69eaa749b9548c/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:01.168928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 972/1971] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:01 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:01.171362 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64840ca0-c92d-4f38-80f0-6eff6e34225f None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:01.185380 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290cc40b-2bd3-414b-ad0b-04a5e97e1941 admin admin] RBAC: Authorizing `identity:create_grant(project_id=403ed8c254684a8e87ee70b92a2dbf11, user_id=029d1350d2ec488c8a69eaa749b9548c, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:01.188301 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290cc40b-2bd3-414b-ad0b-04a5e97e1941 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:01.188634 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64840ca0-c92d-4f38-80f0-6eff6e34225f None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:01.194519 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449b2273-34d7-4390-8090-24d180b33062 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:01.195138 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449b2273-34d7-4390-8090-24d180b33062 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:01.195428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449b2273-34d7-4390-8090-24d180b33062 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:01.195690 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449b2273-34d7-4390-8090-24d180b33062 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:01.196413 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449b2273-34d7-4390-8090-24d180b33062 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:01.198357 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449b2273-34d7-4390-8090-24d180b33062 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:01.271621 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1000/1972] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:01 2022] PUT /identity/v3/projects/403ed8c254684a8e87ee70b92a2dbf11/users/029d1350d2ec488c8a69eaa749b9548c/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:01.271832 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 973/1973] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:01 2022] POST /identity/v3/users => generated 300 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:01.273955 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290cc40b-2bd3-414b-ad0b-04a5e97e1941 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:01.275699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-290cc40b-2bd3-414b-ad0b-04a5e97e1941 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:01.279168 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449b2273-34d7-4390-8090-24d180b33062 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:01.296263 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449b2273-34d7-4390-8090-24d180b33062 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:01.320317 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:01.352531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:01.379212 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290cc40b-2bd3-414b-ad0b-04a5e97e1941 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:01.385014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5aab53-c543-4583-b82c-a95c0f48b206 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:01.385638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5aab53-c543-4583-b82c-a95c0f48b206 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:01.385915 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5aab53-c543-4583-b82c-a95c0f48b206 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:01.386185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5aab53-c543-4583-b82c-a95c0f48b206 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:01.394907 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da456a22-0950-42cc-8d75-7438cfa6d4f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:01.395637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da456a22-0950-42cc-8d75-7438cfa6d4f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:01.395982 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da456a22-0950-42cc-8d75-7438cfa6d4f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:01.396353 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da456a22-0950-42cc-8d75-7438cfa6d4f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:01.397330 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da456a22-0950-42cc-8d75-7438cfa6d4f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:01.399045 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da456a22-0950-42cc-8d75-7438cfa6d4f3 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:01.409953 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 974/1974] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:01 2022] GET /identity/v3/roles => generated 1420 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:03:01.410854 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da456a22-0950-42cc-8d75-7438cfa6d4f3 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:01.420967 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da456a22-0950-42cc-8d75-7438cfa6d4f3 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:01.426704 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e11a536-5434-4444-860e-1e8ff4938b28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:01.427288 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e11a536-5434-4444-860e-1e8ff4938b28 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:01.427569 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e11a536-5434-4444-860e-1e8ff4938b28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:01.427857 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e11a536-5434-4444-860e-1e8ff4938b28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:01.428475 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e11a536-5434-4444-860e-1e8ff4938b28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:01.430053 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e11a536-5434-4444-860e-1e8ff4938b28 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:01.440571 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 975/1975] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:01 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:01.441367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e11a536-5434-4444-860e-1e8ff4938b28 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:01.451203 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e11a536-5434-4444-860e-1e8ff4938b28 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:01.453817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5aab53-c543-4583-b82c-a95c0f48b206 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:01.455308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5aab53-c543-4583-b82c-a95c0f48b206 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:01.456097 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5b8861-da49-442b-b72d-7a4d6c946b68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:01.456912 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5b8861-da49-442b-b72d-7a4d6c946b68 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:01.457194 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5b8861-da49-442b-b72d-7a4d6c946b68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:01.457467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5b8861-da49-442b-b72d-7a4d6c946b68 admin admin] PATH_INFO: `/v3/projects/403ed8c254684a8e87ee70b92a2dbf11/users/029d1350d2ec488c8a69eaa749b9548c/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:01.460141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5b8861-da49-442b-b72d-7a4d6c946b68 admin admin] RBAC: Authorizing `identity:create_grant(project_id=403ed8c254684a8e87ee70b92a2dbf11, user_id=029d1350d2ec488c8a69eaa749b9548c, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:01.463132 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d5b8861-da49-442b-b72d-7a4d6c946b68 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:01.471323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1001/1976] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:01 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:01.471963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5aab53-c543-4583-b82c-a95c0f48b206 service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:01.477689 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 976/1977] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:01 2022] PUT /identity/v3/projects/403ed8c254684a8e87ee70b92a2dbf11/users/029d1350d2ec488c8a69eaa749b9548c/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:01.490382 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3505ba18-32d8-486e-b521-aa01c016d04d None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:01.490677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3505ba18-32d8-486e-b521-aa01c016d04d None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:01.491007 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3505ba18-32d8-486e-b521-aa01c016d04d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:01.491672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5aab53-c543-4583-b82c-a95c0f48b206 service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:01.527381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-3505ba18-32d8-486e-b521-aa01c016d04d None None] MFA Rules not processed for user `029d1350d2ec488c8a69eaa749b9548c`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:01.540134 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3505ba18-32d8-486e-b521-aa01c016d04d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:01.620486 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:01.639856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63a7d17-6c2e-4b93-92fd-56706bd27042 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:01.641642 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63a7d17-6c2e-4b93-92fd-56706bd27042 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:01.642922 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63a7d17-6c2e-4b93-92fd-56706bd27042 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:01.644883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63a7d17-6c2e-4b93-92fd-56706bd27042 admin admin] PATH_INFO: `/v3/projects/df1ff703fc014f5fb2e1f918ae48ea08/users/b214c949bfdd443ea1e0860d630e759d/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:01.651705 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:01.687609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 977/1978] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:01 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:01.695335 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3505ba18-32d8-486e-b521-aa01c016d04d None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:01.701267 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63a7d17-6c2e-4b93-92fd-56706bd27042 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df1ff703fc014f5fb2e1f918ae48ea08, user_id=b214c949bfdd443ea1e0860d630e759d, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:01.708406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63a7d17-6c2e-4b93-92fd-56706bd27042 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:01.716890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3505ba18-32d8-486e-b521-aa01c016d04d None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:01.724426 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f5a22f-f3f9-4abb-a5fd-3ee6516e9591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:01.728857 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f5a22f-f3f9-4abb-a5fd-3ee6516e9591 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:01.728857 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f5a22f-f3f9-4abb-a5fd-3ee6516e9591 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:01.728857 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f5a22f-f3f9-4abb-a5fd-3ee6516e9591 admin admin] PATH_INFO: `/v3/users/8e60004b6011499fb5a914bedc7b47a3` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:01.728857 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f5a22f-f3f9-4abb-a5fd-3ee6516e9591 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8e60004b6011499fb5a914bedc7b47a3)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:01.731878 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f5a22f-f3f9-4abb-a5fd-3ee6516e9591 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:01.785247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1002/1979] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:01 2022] PUT /identity/v3/projects/df1ff703fc014f5fb2e1f918ae48ea08/users/b214c949bfdd443ea1e0860d630e759d/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:01.786702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63a7d17-6c2e-4b93-92fd-56706bd27042 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:01.788810 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d63a7d17-6c2e-4b93-92fd-56706bd27042 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:01.892748 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63a7d17-6c2e-4b93-92fd-56706bd27042 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:01.896817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f7120b-548b-4dce-8bda-ed8192c7da3f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:01.897426 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f7120b-548b-4dce-8bda-ed8192c7da3f service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:01.897706 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f7120b-548b-4dce-8bda-ed8192c7da3f service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:01.897969 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f7120b-548b-4dce-8bda-ed8192c7da3f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:01.966111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:01.988170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e8f5a22f-f3f9-4abb-a5fd-3ee6516e9591 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8e60004b6011499fb5a914bedc7b47a3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:02.011751 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f7120b-548b-4dce-8bda-ed8192c7da3f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:02.013230 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f7120b-548b-4dce-8bda-ed8192c7da3f service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:02.025265 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1003/1980] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:01 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:02.026114 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f7120b-548b-4dce-8bda-ed8192c7da3f service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:02.137984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f7120b-548b-4dce-8bda-ed8192c7da3f service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:02.145120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bdecece-2cd1-435c-af2b-242d30ec023e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:02.146874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdecece-2cd1-435c-af2b-242d30ec023e admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:02.146993 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e8f5a22f-f3f9-4abb-a5fd-3ee6516e9591 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8e60004b6011499fb5a914bedc7b47a3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:02.147087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdecece-2cd1-435c-af2b-242d30ec023e admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:02.147087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bdecece-2cd1-435c-af2b-242d30ec023e admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:02.147247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bdecece-2cd1-435c-af2b-242d30ec023e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:02.148780 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bdecece-2cd1-435c-af2b-242d30ec023e admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:02.149186 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e8f5a22f-f3f9-4abb-a5fd-3ee6516e9591 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8e60004b6011499fb5a914bedc7b47a3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:02.160533 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1004/1981] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:02 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:02.161183 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bdecece-2cd1-435c-af2b-242d30ec023e admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:02.163714 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-4bdecece-2cd1-435c-af2b-242d30ec023e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:02.176380 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e8f5a22f-f3f9-4abb-a5fd-3ee6516e9591 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8e60004b6011499fb5a914bedc7b47a3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:02.179690 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bdecece-2cd1-435c-af2b-242d30ec023e admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:02.184185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-167b5b7f-4eb0-43e5-8585-81d7ef112679 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:02.184798 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167b5b7f-4eb0-43e5-8585-81d7ef112679 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:02.185078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167b5b7f-4eb0-43e5-8585-81d7ef112679 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:02.185177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e8f5a22f-f3f9-4abb-a5fd-3ee6516e9591 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8e60004b6011499fb5a914bedc7b47a3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:02.185341 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-167b5b7f-4eb0-43e5-8585-81d7ef112679 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:02.188768 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-167b5b7f-4eb0-43e5-8585-81d7ef112679 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:02.222277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 978/1982] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:01 2022] DELETE /identity/v3/users/8e60004b6011499fb5a914bedc7b47a3 => generated 0 bytes in 533 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:02.222277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f5a22f-f3f9-4abb-a5fd-3ee6516e9591 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:02.223098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e8f5a22f-f3f9-4abb-a5fd-3ee6516e9591 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:02.275947 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-167b5b7f-4eb0-43e5-8585-81d7ef112679 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:02.277229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-167b5b7f-4eb0-43e5-8585-81d7ef112679 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:02.277366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=98,keystone:DELETE=1,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:02.298456 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1005/1983] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:02 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:02.300725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c3804c-a4e8-4fbe-a7f6-19deee743ac7 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:02.301455 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c3804c-a4e8-4fbe-a7f6-19deee743ac7 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:02.302182 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c3804c-a4e8-4fbe-a7f6-19deee743ac7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:02.329706 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-14c3804c-a4e8-4fbe-a7f6-19deee743ac7 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:02.331715 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-14c3804c-a4e8-4fbe-a7f6-19deee743ac7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:02.348916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f5a22f-f3f9-4abb-a5fd-3ee6516e9591 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:02.351503 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1006/1984] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:02.352437 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c3804c-a4e8-4fbe-a7f6-19deee743ac7 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:02.353924 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-14c3804c-a4e8-4fbe-a7f6-19deee743ac7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:02.362838 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f163fcff-0b15-45bc-ac01-534660d5e8a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:02.363951 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f163fcff-0b15-45bc-ac01-534660d5e8a2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:02.364633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f163fcff-0b15-45bc-ac01-534660d5e8a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:02.364907 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f163fcff-0b15-45bc-ac01-534660d5e8a2 admin admin] PATH_INFO: `/v3/projects/df1ff703fc014f5fb2e1f918ae48ea08/users/b214c949bfdd443ea1e0860d630e759d/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:02.368133 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f163fcff-0b15-45bc-ac01-534660d5e8a2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=df1ff703fc014f5fb2e1f918ae48ea08, user_id=b214c949bfdd443ea1e0860d630e759d, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:02.371946 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c3804c-a4e8-4fbe-a7f6-19deee743ac7 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:02.372629 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f163fcff-0b15-45bc-ac01-534660d5e8a2 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:02.382157 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72834ed-06c2-4eed-bc2a-73c08205849b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:02.383535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72834ed-06c2-4eed-bc2a-73c08205849b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:02.383873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72834ed-06c2-4eed-bc2a-73c08205849b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:02.384256 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72834ed-06c2-4eed-bc2a-73c08205849b admin admin] PATH_INFO: `/v3/users/f294eecaf0ee4fbf9f70a32780f950dd` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:02.385605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72834ed-06c2-4eed-bc2a-73c08205849b admin admin] RBAC: Authorizing `identity:delete_user(user_id=f294eecaf0ee4fbf9f70a32780f950dd)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:02.388019 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 979/1985] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:02 2022] PUT /identity/v3/projects/df1ff703fc014f5fb2e1f918ae48ea08/users/b214c949bfdd443ea1e0860d630e759d/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:02.388212 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72834ed-06c2-4eed-bc2a-73c08205849b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:02.389892 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f163fcff-0b15-45bc-ac01-534660d5e8a2 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:02.393412 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f163fcff-0b15-45bc-ac01-534660d5e8a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:02.468127 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f163fcff-0b15-45bc-ac01-534660d5e8a2 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:02.472674 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a227dcd-84d6-44b0-9f93-452c0be8bd34 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:02.473341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a227dcd-84d6-44b0-9f93-452c0be8bd34 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:02.473619 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a227dcd-84d6-44b0-9f93-452c0be8bd34 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:02.473889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a227dcd-84d6-44b0-9f93-452c0be8bd34 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:02.538254 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a227dcd-84d6-44b0-9f93-452c0be8bd34 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:02.539745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a227dcd-84d6-44b0-9f93-452c0be8bd34 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:02.551375 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 980/1986] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:02.552353 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91ec18f-05b8-4cf3-a5f6-c2e862c5a1e4 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:02.552954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91ec18f-05b8-4cf3-a5f6-c2e862c5a1e4 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:02.553387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91ec18f-05b8-4cf3-a5f6-c2e862c5a1e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:02.591652 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-d91ec18f-05b8-4cf3-a5f6-c2e862c5a1e4 None None] MFA Rules not processed for user `b214c949bfdd443ea1e0860d630e759d`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:02.592789 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d91ec18f-05b8-4cf3-a5f6-c2e862c5a1e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:02.606223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e72834ed-06c2-4eed-bc2a-73c08205849b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f294eecaf0ee4fbf9f70a32780f950dd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:02.680330 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 981/1987] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:02 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:02.695947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91ec18f-05b8-4cf3-a5f6-c2e862c5a1e4 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:02.697242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d91ec18f-05b8-4cf3-a5f6-c2e862c5a1e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:02.709626 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e72834ed-06c2-4eed-bc2a-73c08205849b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f294eecaf0ee4fbf9f70a32780f950dd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:02.749018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e72834ed-06c2-4eed-bc2a-73c08205849b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f294eecaf0ee4fbf9f70a32780f950dd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:02.749786 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e72834ed-06c2-4eed-bc2a-73c08205849b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f294eecaf0ee4fbf9f70a32780f950dd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:02.758083 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e72834ed-06c2-4eed-bc2a-73c08205849b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f294eecaf0ee4fbf9f70a32780f950dd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:02.769705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1007/1988] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:02 2022] DELETE /identity/v3/users/f294eecaf0ee4fbf9f70a32780f950dd => generated 0 bytes in 414 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:02.771264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45334c29-d71a-44ca-9578-9f7c8a89cfe0 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:02.771552 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45334c29-d71a-44ca-9578-9f7c8a89cfe0 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:02.771853 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45334c29-d71a-44ca-9578-9f7c8a89cfe0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:02.779253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91ec18f-05b8-4cf3-a5f6-c2e862c5a1e4 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:02.791746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-45334c29-d71a-44ca-9578-9f7c8a89cfe0 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:02.793140 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-45334c29-d71a-44ca-9578-9f7c8a89cfe0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:02.823573 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:02.861323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf78165-cf13-42ce-b7d4-5f6ef38e15a9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:02.861323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf78165-cf13-42ce-b7d4-5f6ef38e15a9 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:02.861323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf78165-cf13-42ce-b7d4-5f6ef38e15a9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:02.861323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf78165-cf13-42ce-b7d4-5f6ef38e15a9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:02.865469 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2cf78165-cf13-42ce-b7d4-5f6ef38e15a9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:02.887997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1008/1989] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:02.928819 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45334c29-d71a-44ca-9578-9f7c8a89cfe0 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:02.930258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-45334c29-d71a-44ca-9578-9f7c8a89cfe0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:02.987719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf78165-cf13-42ce-b7d4-5f6ef38e15a9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:02.990843 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf78165-cf13-42ce-b7d4-5f6ef38e15a9 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:03.005972 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 982/1990] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:02 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:03.034303 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45334c29-d71a-44ca-9578-9f7c8a89cfe0 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:03.040153 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26708b9b-b8eb-4f77-a0c0-80e50b20054b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:03.041751 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26708b9b-b8eb-4f77-a0c0-80e50b20054b service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:03.042066 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26708b9b-b8eb-4f77-a0c0-80e50b20054b service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:03.042370 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26708b9b-b8eb-4f77-a0c0-80e50b20054b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:03.055022 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26708b9b-b8eb-4f77-a0c0-80e50b20054b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:03.056548 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26708b9b-b8eb-4f77-a0c0-80e50b20054b service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:03.070070 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1009/1991] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:03:03.241365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf78165-cf13-42ce-b7d4-5f6ef38e15a9 service cinder] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:03.242856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2cf78165-cf13-42ce-b7d4-5f6ef38e15a9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:03.262143 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf78165-cf13-42ce-b7d4-5f6ef38e15a9 service cinder] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:03.267958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4b4d67-eb36-4efa-b2e5-fd678164b9ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:03.268977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4b4d67-eb36-4efa-b2e5-fd678164b9ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:03.269435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4b4d67-eb36-4efa-b2e5-fd678164b9ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:03.269874 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4b4d67-eb36-4efa-b2e5-fd678164b9ec admin admin] PATH_INFO: `/v3/users/153768c660764af6af76fe7a6b23a68a` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:03.271494 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4b4d67-eb36-4efa-b2e5-fd678164b9ec admin admin] RBAC: Authorizing `identity:delete_user(user_id=153768c660764af6af76fe7a6b23a68a)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:03.273986 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4b4d67-eb36-4efa-b2e5-fd678164b9ec admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:03.288723 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26708b9b-b8eb-4f77-a0c0-80e50b20054b service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:03.289916 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-26708b9b-b8eb-4f77-a0c0-80e50b20054b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:03.306997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26708b9b-b8eb-4f77-a0c0-80e50b20054b service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:03.312336 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa745cc-4aab-41e5-8f86-9e365b186504 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:03.313432 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa745cc-4aab-41e5-8f86-9e365b186504 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:03.313812 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa745cc-4aab-41e5-8f86-9e365b186504 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:03.314281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa745cc-4aab-41e5-8f86-9e365b186504 admin admin] PATH_INFO: `/v3/projects/02c99699ea244fee9ad03b9f12c512dc` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:03.315759 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa745cc-4aab-41e5-8f86-9e365b186504 admin admin] RBAC: Authorizing `identity:delete_project(project_id=02c99699ea244fee9ad03b9f12c512dc)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:03.318550 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa745cc-4aab-41e5-8f86-9e365b186504 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:03.377660 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:03.557172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-ef4b4d67-eb36-4efa-b2e5-fd678164b9ec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '153768c660764af6af76fe7a6b23a68a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:03.557653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9aa745cc-4aab-41e5-8f86-9e365b186504 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '02c99699ea244fee9ad03b9f12c512dc', 'request_id': 'req-9aa745cc-4aab-41e5-8f86-9e365b186504'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:03.642203 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9aa745cc-4aab-41e5-8f86-9e365b186504 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '02c99699ea244fee9ad03b9f12c512dc', 'request_id': 'req-9aa745cc-4aab-41e5-8f86-9e365b186504'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:03.651622 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9aa745cc-4aab-41e5-8f86-9e365b186504 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '02c99699ea244fee9ad03b9f12c512dc', 'request_id': 'req-9aa745cc-4aab-41e5-8f86-9e365b186504'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:03.664594 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9aa745cc-4aab-41e5-8f86-9e365b186504 admin admin] The token cache is being invalidate because project 02c99699ea244fee9ad03b9f12c512dc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:03.664887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9aa745cc-4aab-41e5-8f86-9e365b186504 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:03.665538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9aa745cc-4aab-41e5-8f86-9e365b186504 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:03.667330 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1010/1992] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:03 2022] DELETE /identity/v3/projects/02c99699ea244fee9ad03b9f12c512dc => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:03.670813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa745cc-4aab-41e5-8f86-9e365b186504 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:03.672162 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-9aa745cc-4aab-41e5-8f86-9e365b186504 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:03.737163 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-ef4b4d67-eb36-4efa-b2e5-fd678164b9ec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '153768c660764af6af76fe7a6b23a68a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:03.737798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-ef4b4d67-eb36-4efa-b2e5-fd678164b9ec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '153768c660764af6af76fe7a6b23a68a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:03.759595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-ef4b4d67-eb36-4efa-b2e5-fd678164b9ec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '153768c660764af6af76fe7a6b23a68a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:03.770626 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa745cc-4aab-41e5-8f86-9e365b186504 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:03.776078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-598f573d-141a-4021-9860-10b73d5c4087 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:03.776410 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-ef4b4d67-eb36-4efa-b2e5-fd678164b9ec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '153768c660764af6af76fe7a6b23a68a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:03.776909 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598f573d-141a-4021-9860-10b73d5c4087 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:03.777193 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598f573d-141a-4021-9860-10b73d5c4087 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:03.777465 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-598f573d-141a-4021-9860-10b73d5c4087 admin admin] PATH_INFO: `/v3/users/8152d89a5a634b698869692856bbbbb6` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:03.778882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598f573d-141a-4021-9860-10b73d5c4087 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8152d89a5a634b698869692856bbbbb6)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:03.780879 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-598f573d-141a-4021-9860-10b73d5c4087 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:03.803315 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 983/1993] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:03 2022] DELETE /identity/v3/users/153768c660764af6af76fe7a6b23a68a => generated 0 bytes in 559 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:03.803315 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4b4d67-eb36-4efa-b2e5-fd678164b9ec admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:03.804838 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ef4b4d67-eb36-4efa-b2e5-fd678164b9ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:03.881340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4b4d67-eb36-4efa-b2e5-fd678164b9ec admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:03.888113 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912578f8-bb9e-4b6f-9f0f-69f3b042bf0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:03.888904 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912578f8-bb9e-4b6f-9f0f-69f3b042bf0d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:03.889292 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912578f8-bb9e-4b6f-9f0f-69f3b042bf0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:03.889778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912578f8-bb9e-4b6f-9f0f-69f3b042bf0d admin admin] PATH_INFO: `/v3/projects/fdc51b455c8b418e81449c8488848521` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:03.896593 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912578f8-bb9e-4b6f-9f0f-69f3b042bf0d admin admin] RBAC: Authorizing `identity:delete_project(project_id=fdc51b455c8b418e81449c8488848521)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:03.897399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912578f8-bb9e-4b6f-9f0f-69f3b042bf0d admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:03.976172 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-598f573d-141a-4021-9860-10b73d5c4087 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8152d89a5a634b698869692856bbbbb6'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:03.998796 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-912578f8-bb9e-4b6f-9f0f-69f3b042bf0d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fdc51b455c8b418e81449c8488848521', 'request_id': 'req-912578f8-bb9e-4b6f-9f0f-69f3b042bf0d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:04.040288 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=84,keystone:DELETE=12,keystone:INSERT=4 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:04.063021 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-598f573d-141a-4021-9860-10b73d5c4087 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8152d89a5a634b698869692856bbbbb6'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:04.090994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-598f573d-141a-4021-9860-10b73d5c4087 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8152d89a5a634b698869692856bbbbb6'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:04.093376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-598f573d-141a-4021-9860-10b73d5c4087 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8152d89a5a634b698869692856bbbbb6'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:04.100428 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-598f573d-141a-4021-9860-10b73d5c4087 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8152d89a5a634b698869692856bbbbb6'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:04.112368 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1011/1994] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:03 2022] DELETE /identity/v3/users/8152d89a5a634b698869692856bbbbb6 => generated 0 bytes in 437 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:04.112368 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69912d01-7b6e-43e2-98d8-6b8803b102b6 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:04.112368 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69912d01-7b6e-43e2-98d8-6b8803b102b6 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:04.112368 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69912d01-7b6e-43e2-98d8-6b8803b102b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:04.131491 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-912578f8-bb9e-4b6f-9f0f-69f3b042bf0d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fdc51b455c8b418e81449c8488848521', 'request_id': 'req-912578f8-bb9e-4b6f-9f0f-69f3b042bf0d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:04.138810 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-912578f8-bb9e-4b6f-9f0f-69f3b042bf0d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fdc51b455c8b418e81449c8488848521', 'request_id': 'req-912578f8-bb9e-4b6f-9f0f-69f3b042bf0d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:04.139576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-69912d01-7b6e-43e2-98d8-6b8803b102b6 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:04.147169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-69912d01-7b6e-43e2-98d8-6b8803b102b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:04.157886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:04.169380 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-912578f8-bb9e-4b6f-9f0f-69f3b042bf0d admin admin] The token cache is being invalidate because project fdc51b455c8b418e81449c8488848521 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:04.170226 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-912578f8-bb9e-4b6f-9f0f-69f3b042bf0d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:04.171135 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-912578f8-bb9e-4b6f-9f0f-69f3b042bf0d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:04.177424 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 984/1995] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:03 2022] DELETE /identity/v3/projects/fdc51b455c8b418e81449c8488848521 => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:04.181582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912578f8-bb9e-4b6f-9f0f-69f3b042bf0d admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:04.183325 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-912578f8-bb9e-4b6f-9f0f-69f3b042bf0d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:04.298918 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1012/1996] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:04.320822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912578f8-bb9e-4b6f-9f0f-69f3b042bf0d admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:04.340774 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4576a9f6-3c61-40b6-b836-f831ddf34819 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:04.341566 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4576a9f6-3c61-40b6-b836-f831ddf34819 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:04.341801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4576a9f6-3c61-40b6-b836-f831ddf34819 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:04.342079 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4576a9f6-3c61-40b6-b836-f831ddf34819 admin admin] PATH_INFO: `/v3/users/05fc5bc8a32042608a32cafe4de085bd` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:04.345034 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4576a9f6-3c61-40b6-b836-f831ddf34819 admin admin] RBAC: Authorizing `identity:delete_user(user_id=05fc5bc8a32042608a32cafe4de085bd)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:04.347929 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4576a9f6-3c61-40b6-b836-f831ddf34819 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:04.430884 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69912d01-7b6e-43e2-98d8-6b8803b102b6 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:04.432567 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-69912d01-7b6e-43e2-98d8-6b8803b102b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:04.555637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69912d01-7b6e-43e2-98d8-6b8803b102b6 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:04.586889 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ffc9ac-c4a8-4934-af02-478cb245c017 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:04.590189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ffc9ac-c4a8-4934-af02-478cb245c017 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:04.591818 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ffc9ac-c4a8-4934-af02-478cb245c017 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:04.592148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46ffc9ac-c4a8-4934-af02-478cb245c017 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:04.617146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ffc9ac-c4a8-4934-af02-478cb245c017 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:04.618967 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46ffc9ac-c4a8-4934-af02-478cb245c017 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:04.621916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-4576a9f6-3c61-40b6-b836-f831ddf34819 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '05fc5bc8a32042608a32cafe4de085bd'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:04.676694 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1013/1997] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:03:04.748939 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-4576a9f6-3c61-40b6-b836-f831ddf34819 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '05fc5bc8a32042608a32cafe4de085bd'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:04.749880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-4576a9f6-3c61-40b6-b836-f831ddf34819 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '05fc5bc8a32042608a32cafe4de085bd'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:04.767428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-4576a9f6-3c61-40b6-b836-f831ddf34819 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '05fc5bc8a32042608a32cafe4de085bd'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:04.787241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-4576a9f6-3c61-40b6-b836-f831ddf34819 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '05fc5bc8a32042608a32cafe4de085bd'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:04.798514 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 985/1998] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:04 2022] DELETE /identity/v3/users/05fc5bc8a32042608a32cafe4de085bd => generated 0 bytes in 617 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:05.183314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:05.214574 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ffc9ac-c4a8-4934-af02-478cb245c017 service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:05.216004 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-46ffc9ac-c4a8-4934-af02-478cb245c017 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:05.242682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4576a9f6-3c61-40b6-b836-f831ddf34819 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:05.244203 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-4576a9f6-3c61-40b6-b836-f831ddf34819 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:05.248724 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46ffc9ac-c4a8-4934-af02-478cb245c017 service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:05.292018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:DELETE=5,keystone:SELECT=93,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:05.369279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e62dc71-2667-411a-b7ee-0afd1b2887c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:05.369931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e62dc71-2667-411a-b7ee-0afd1b2887c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:05.370248 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e62dc71-2667-411a-b7ee-0afd1b2887c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:05.370528 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e62dc71-2667-411a-b7ee-0afd1b2887c1 admin admin] PATH_INFO: `/v3/projects/62db6dcb146444f997929364e3ba7fd7` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:05.371838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e62dc71-2667-411a-b7ee-0afd1b2887c1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=62db6dcb146444f997929364e3ba7fd7)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:05.372085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4576a9f6-3c61-40b6-b836-f831ddf34819 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:05.374480 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e62dc71-2667-411a-b7ee-0afd1b2887c1 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:05.378999 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f59a6fb-9a14-4ed7-98ca-21d777012b34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:05.380003 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f59a6fb-9a14-4ed7-98ca-21d777012b34 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:05.380495 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f59a6fb-9a14-4ed7-98ca-21d777012b34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:05.381220 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f59a6fb-9a14-4ed7-98ca-21d777012b34 admin admin] PATH_INFO: `/v3/users/3ca42b8753dc40f7a9127f6582298770` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:05.382749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f59a6fb-9a14-4ed7-98ca-21d777012b34 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3ca42b8753dc40f7a9127f6582298770)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:05.385182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f59a6fb-9a14-4ed7-98ca-21d777012b34 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:05.571581 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-4e62dc71-2667-411a-b7ee-0afd1b2887c1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '62db6dcb146444f997929364e3ba7fd7', 'request_id': 'req-4e62dc71-2667-411a-b7ee-0afd1b2887c1'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:05.573311 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2f59a6fb-9a14-4ed7-98ca-21d777012b34 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3ca42b8753dc40f7a9127f6582298770'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:05.640049 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-4e62dc71-2667-411a-b7ee-0afd1b2887c1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '62db6dcb146444f997929364e3ba7fd7', 'request_id': 'req-4e62dc71-2667-411a-b7ee-0afd1b2887c1'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:05.647971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-4e62dc71-2667-411a-b7ee-0afd1b2887c1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '62db6dcb146444f997929364e3ba7fd7', 'request_id': 'req-4e62dc71-2667-411a-b7ee-0afd1b2887c1'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:05.674773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-4e62dc71-2667-411a-b7ee-0afd1b2887c1 admin admin] The token cache is being invalidate because project 62db6dcb146444f997929364e3ba7fd7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:05.674773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-4e62dc71-2667-411a-b7ee-0afd1b2887c1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:05.674773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-4e62dc71-2667-411a-b7ee-0afd1b2887c1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:05.674773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1014/1999] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:05 2022] DELETE /identity/v3/projects/62db6dcb146444f997929364e3ba7fd7 => generated 0 bytes in 460 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:05.675455 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e62dc71-2667-411a-b7ee-0afd1b2887c1 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:05.677028 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-4e62dc71-2667-411a-b7ee-0afd1b2887c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:05.692345 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2f59a6fb-9a14-4ed7-98ca-21d777012b34 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3ca42b8753dc40f7a9127f6582298770'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:05.702980 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2f59a6fb-9a14-4ed7-98ca-21d777012b34 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3ca42b8753dc40f7a9127f6582298770'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:05.719746 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2f59a6fb-9a14-4ed7-98ca-21d777012b34 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3ca42b8753dc40f7a9127f6582298770'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:05.738718 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2f59a6fb-9a14-4ed7-98ca-21d777012b34 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3ca42b8753dc40f7a9127f6582298770'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:05.748947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 986/2000] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:05 2022] DELETE /identity/v3/users/3ca42b8753dc40f7a9127f6582298770 => generated 0 bytes in 504 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:05.749881 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f59a6fb-9a14-4ed7-98ca-21d777012b34 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:05.756177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2f59a6fb-9a14-4ed7-98ca-21d777012b34 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:05.784793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e62dc71-2667-411a-b7ee-0afd1b2887c1 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:05.802172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:05.852687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb8b42d-0a96-404b-9f1c-486a114e1074 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:05.853494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb8b42d-0a96-404b-9f1c-486a114e1074 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:05.853779 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb8b42d-0a96-404b-9f1c-486a114e1074 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:05.854052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb8b42d-0a96-404b-9f1c-486a114e1074 admin admin] PATH_INFO: `/v3/projects/0ccb60fdca144211a380fd193c8034b1` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:05.857607 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb8b42d-0a96-404b-9f1c-486a114e1074 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ccb60fdca144211a380fd193c8034b1)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:05.859461 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f59a6fb-9a14-4ed7-98ca-21d777012b34 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:05.859821 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb8b42d-0a96-404b-9f1c-486a114e1074 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:05.872345 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e69ae0-84cf-44c1-a791-5aef03a63fcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:05.873598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e69ae0-84cf-44c1-a791-5aef03a63fcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:05.873882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e69ae0-84cf-44c1-a791-5aef03a63fcb admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:05.874163 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e69ae0-84cf-44c1-a791-5aef03a63fcb admin admin] PATH_INFO: `/v3/projects/537ad5add80d45e59823a73b177657d1` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:05.878286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e69ae0-84cf-44c1-a791-5aef03a63fcb admin admin] RBAC: Authorizing `identity:delete_project(project_id=537ad5add80d45e59823a73b177657d1)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:05.880387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e69ae0-84cf-44c1-a791-5aef03a63fcb admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:06.001106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-deb8b42d-0a96-404b-9f1c-486a114e1074 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ccb60fdca144211a380fd193c8034b1', 'request_id': 'req-deb8b42d-0a96-404b-9f1c-486a114e1074'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:06.001562 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-24e69ae0-84cf-44c1-a791-5aef03a63fcb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '537ad5add80d45e59823a73b177657d1', 'request_id': 'req-24e69ae0-84cf-44c1-a791-5aef03a63fcb'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:06.119847 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-deb8b42d-0a96-404b-9f1c-486a114e1074 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ccb60fdca144211a380fd193c8034b1', 'request_id': 'req-deb8b42d-0a96-404b-9f1c-486a114e1074'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:06.126466 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-deb8b42d-0a96-404b-9f1c-486a114e1074 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ccb60fdca144211a380fd193c8034b1', 'request_id': 'req-deb8b42d-0a96-404b-9f1c-486a114e1074'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:06.140544 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-deb8b42d-0a96-404b-9f1c-486a114e1074 admin admin] The token cache is being invalidate because project 0ccb60fdca144211a380fd193c8034b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:06.140544 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-deb8b42d-0a96-404b-9f1c-486a114e1074 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:06.140544 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-deb8b42d-0a96-404b-9f1c-486a114e1074 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:06.142322 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1015/2001] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:05 2022] DELETE /identity/v3/projects/0ccb60fdca144211a380fd193c8034b1 => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:06.144125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8621bcb-1aa5-4a5e-b720-628cdfad91fb None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:06.144412 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8621bcb-1aa5-4a5e-b720-628cdfad91fb None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:06.144688 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8621bcb-1aa5-4a5e-b720-628cdfad91fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:06.168249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c8621bcb-1aa5-4a5e-b720-628cdfad91fb None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:06.171376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c8621bcb-1aa5-4a5e-b720-628cdfad91fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:06.193908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-24e69ae0-84cf-44c1-a791-5aef03a63fcb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '537ad5add80d45e59823a73b177657d1', 'request_id': 'req-24e69ae0-84cf-44c1-a791-5aef03a63fcb'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:06.200549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-24e69ae0-84cf-44c1-a791-5aef03a63fcb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '537ad5add80d45e59823a73b177657d1', 'request_id': 'req-24e69ae0-84cf-44c1-a791-5aef03a63fcb'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:06.212992 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-24e69ae0-84cf-44c1-a791-5aef03a63fcb admin admin] The token cache is being invalidate because project 537ad5add80d45e59823a73b177657d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:06.213687 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-24e69ae0-84cf-44c1-a791-5aef03a63fcb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:06.214573 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-24e69ae0-84cf-44c1-a791-5aef03a63fcb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:06.216396 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 987/2002] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:05 2022] DELETE /identity/v3/projects/537ad5add80d45e59823a73b177657d1 => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:06.218516 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d479e49-d60e-49a4-93dc-51d1d6308788 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:06.218799 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d479e49-d60e-49a4-93dc-51d1d6308788 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:06.219069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d479e49-d60e-49a4-93dc-51d1d6308788 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:06.240385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1016/2003] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:06.241676 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652a64e9-2e14-40a6-9478-24b02a1f722e None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:06.241957 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652a64e9-2e14-40a6-9478-24b02a1f722e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:06.242224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652a64e9-2e14-40a6-9478-24b02a1f722e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:06.247598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-1d479e49-d60e-49a4-93dc-51d1d6308788 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:06.248867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-1d479e49-d60e-49a4-93dc-51d1d6308788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:06.257128 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 988/2004] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:06.260389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d479e49-d60e-49a4-93dc-51d1d6308788 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:06.260389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-1d479e49-d60e-49a4-93dc-51d1d6308788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:06.269118 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-652a64e9-2e14-40a6-9478-24b02a1f722e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:06.269118 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-652a64e9-2e14-40a6-9478-24b02a1f722e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:06.271832 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d479e49-d60e-49a4-93dc-51d1d6308788 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:06.275047 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1017/2005] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:06.276228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d43f7f0-b4b5-4d0f-af36-d5f8b3703fa6 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:06.276454 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ec6acc-3b71-4fcc-94af-6197bb0ebcd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:06.276604 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d43f7f0-b4b5-4d0f-af36-d5f8b3703fa6 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:06.276800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d43f7f0-b4b5-4d0f-af36-d5f8b3703fa6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:06.277131 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ec6acc-3b71-4fcc-94af-6197bb0ebcd2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:06.277413 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ec6acc-3b71-4fcc-94af-6197bb0ebcd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:06.277707 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ec6acc-3b71-4fcc-94af-6197bb0ebcd2 admin admin] PATH_INFO: `/v3/users/b214c949bfdd443ea1e0860d630e759d` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:06.278880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ec6acc-3b71-4fcc-94af-6197bb0ebcd2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b214c949bfdd443ea1e0860d630e759d)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:06.280803 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ec6acc-3b71-4fcc-94af-6197bb0ebcd2 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:06.298730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-7d43f7f0-b4b5-4d0f-af36-d5f8b3703fa6 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:06.300744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-7d43f7f0-b4b5-4d0f-af36-d5f8b3703fa6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:06.309249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1018/2006] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:06.309846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d43f7f0-b4b5-4d0f-af36-d5f8b3703fa6 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:06.311103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-7d43f7f0-b4b5-4d0f-af36-d5f8b3703fa6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:06.366795 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d43f7f0-b4b5-4d0f-af36-d5f8b3703fa6 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:06.370521 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff93325-d8d2-416a-b37a-5b9eef7f5b5a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:06.371151 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff93325-d8d2-416a-b37a-5b9eef7f5b5a service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:06.371428 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff93325-d8d2-416a-b37a-5b9eef7f5b5a service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:06.371695 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff93325-d8d2-416a-b37a-5b9eef7f5b5a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:06.380719 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff93325-d8d2-416a-b37a-5b9eef7f5b5a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:06.382154 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff93325-d8d2-416a-b37a-5b9eef7f5b5a service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:06.391581 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1019/2007] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:06.392989 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a5af2c-5818-41bd-ab76-b7db1a94d3c8 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:06.393273 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a5af2c-5818-41bd-ab76-b7db1a94d3c8 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:06.393583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a5af2c-5818-41bd-ab76-b7db1a94d3c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:06.413456 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-75a5af2c-5818-41bd-ab76-b7db1a94d3c8 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:06.414693 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-75a5af2c-5818-41bd-ab76-b7db1a94d3c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:06.425195 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1020/2008] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:06.425195 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a5af2c-5818-41bd-ab76-b7db1a94d3c8 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:06.435048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a5af2c-5818-41bd-ab76-b7db1a94d3c8 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:06.441655 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b8d0da-8f44-4e79-9234-66c4dd2df2c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:06.444238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b8d0da-8f44-4e79-9234-66c4dd2df2c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:06.444238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b8d0da-8f44-4e79-9234-66c4dd2df2c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:06.444238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b8d0da-8f44-4e79-9234-66c4dd2df2c8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:06.444238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b8d0da-8f44-4e79-9234-66c4dd2df2c8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:03:06.444238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b8d0da-8f44-4e79-9234-66c4dd2df2c8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:06.445728 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b8d0da-8f44-4e79-9234-66c4dd2df2c8 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:06.459185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1021/2009] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:03:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:03:06.459753 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b8d0da-8f44-4e79-9234-66c4dd2df2c8 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:06.470207 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b8d0da-8f44-4e79-9234-66c4dd2df2c8 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:06.476864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf94bc1a-7554-4775-a2ac-6f084ba13346 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:06.477449 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf94bc1a-7554-4775-a2ac-6f084ba13346 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:06.477888 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf94bc1a-7554-4775-a2ac-6f084ba13346 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:06.478227 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf94bc1a-7554-4775-a2ac-6f084ba13346 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:06.479221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf94bc1a-7554-4775-a2ac-6f084ba13346 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:03:06.479221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf94bc1a-7554-4775-a2ac-6f084ba13346 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:06.479516 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-31ec6acc-3b71-4fcc-94af-6197bb0ebcd2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b214c949bfdd443ea1e0860d630e759d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:06.480802 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf94bc1a-7554-4775-a2ac-6f084ba13346 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:06.494762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1022/2010] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:03:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:03:06.501599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf94bc1a-7554-4775-a2ac-6f084ba13346 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:06.505674 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf94bc1a-7554-4775-a2ac-6f084ba13346 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:06.510624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b792010-a5ee-498b-a733-4d59967c3921 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:06.511222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b792010-a5ee-498b-a733-4d59967c3921 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:06.511506 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b792010-a5ee-498b-a733-4d59967c3921 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:06.511798 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b792010-a5ee-498b-a733-4d59967c3921 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:06.512410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b792010-a5ee-498b-a733-4d59967c3921 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:06.514420 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b792010-a5ee-498b-a733-4d59967c3921 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:06.582514 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-31ec6acc-3b71-4fcc-94af-6197bb0ebcd2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b214c949bfdd443ea1e0860d630e759d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:06.583060 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-31ec6acc-3b71-4fcc-94af-6197bb0ebcd2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b214c949bfdd443ea1e0860d630e759d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:06.595985 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-31ec6acc-3b71-4fcc-94af-6197bb0ebcd2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b214c949bfdd443ea1e0860d630e759d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:06.604194 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-31ec6acc-3b71-4fcc-94af-6197bb0ebcd2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b214c949bfdd443ea1e0860d630e759d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:06.610215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 989/2011] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:06 2022] DELETE /identity/v3/users/b214c949bfdd443ea1e0860d630e759d => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:06.611023 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ec6acc-3b71-4fcc-94af-6197bb0ebcd2 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:06.612415 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-31ec6acc-3b71-4fcc-94af-6197bb0ebcd2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:06.663180 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1023/2012] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:06 2022] POST /identity/v3/projects => generated 407 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:06.664206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b792010-a5ee-498b-a733-4d59967c3921 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:06.668190 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-2b792010-a5ee-498b-a733-4d59967c3921 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:06.695340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ec6acc-3b71-4fcc-94af-6197bb0ebcd2 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:06.753689 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b792010-a5ee-498b-a733-4d59967c3921 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:06.758092 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6be2070-6ee4-4c54-a0fb-70f134d8cf71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:06.758881 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6be2070-6ee4-4c54-a0fb-70f134d8cf71 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:06.759255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6be2070-6ee4-4c54-a0fb-70f134d8cf71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:06.759616 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6be2070-6ee4-4c54-a0fb-70f134d8cf71 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:06.760316 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6be2070-6ee4-4c54-a0fb-70f134d8cf71 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:06.760639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5446836-27c6-479d-87f5-45b102914c97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:06.761397 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5446836-27c6-479d-87f5-45b102914c97 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:06.761806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5446836-27c6-479d-87f5-45b102914c97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:06.762228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5446836-27c6-479d-87f5-45b102914c97 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:06.762864 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6be2070-6ee4-4c54-a0fb-70f134d8cf71 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:06.763187 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5446836-27c6-479d-87f5-45b102914c97 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:06.765235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5446836-27c6-479d-87f5-45b102914c97 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:06.830876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:INSERT=8,keystone:SELECT=92 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:06.901247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1024/2013] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:06 2022] POST /identity/v3/users => generated 304 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:06.909052 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 990/2014] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:06 2022] POST /identity/v3/projects => generated 365 bytes in 298 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:06.912764 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5446836-27c6-479d-87f5-45b102914c97 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:06.916541 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6be2070-6ee4-4c54-a0fb-70f134d8cf71 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:06.951340 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5446836-27c6-479d-87f5-45b102914c97 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:07.024881 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6328131-d306-45bd-9022-6085ef39db2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:07.025679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6328131-d306-45bd-9022-6085ef39db2e admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:07.025971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6328131-d306-45bd-9022-6085ef39db2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:07.026255 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6328131-d306-45bd-9022-6085ef39db2e admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:07.027022 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6328131-d306-45bd-9022-6085ef39db2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:07.028848 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6328131-d306-45bd-9022-6085ef39db2e admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:07.043129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1025/2015] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:06 2022] GET /identity/v3/roles => generated 1420 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:07.047917 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6328131-d306-45bd-9022-6085ef39db2e admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:07.054720 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6be2070-6ee4-4c54-a0fb-70f134d8cf71 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:07.062416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6328131-d306-45bd-9022-6085ef39db2e admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:07.062877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b039b7-eeb4-4072-8e4a-4acc803ec508 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:07.063638 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b039b7-eeb4-4072-8e4a-4acc803ec508 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:07.068549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b039b7-eeb4-4072-8e4a-4acc803ec508 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:07.068549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b039b7-eeb4-4072-8e4a-4acc803ec508 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:07.068549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b039b7-eeb4-4072-8e4a-4acc803ec508 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:07.069193 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b039b7-eeb4-4072-8e4a-4acc803ec508 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:07.071584 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd25308-bbad-4117-8529-05ba5eb7b767 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:07.072319 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd25308-bbad-4117-8529-05ba5eb7b767 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:07.072675 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd25308-bbad-4117-8529-05ba5eb7b767 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:07.076468 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd25308-bbad-4117-8529-05ba5eb7b767 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:07.077237 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd25308-bbad-4117-8529-05ba5eb7b767 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:07.079209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd25308-bbad-4117-8529-05ba5eb7b767 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:07.092696 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1026/2016] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:07 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:03:07.097757 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd25308-bbad-4117-8529-05ba5eb7b767 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:07.113182 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd25308-bbad-4117-8529-05ba5eb7b767 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:07.115017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4236f9-0cf6-4ba9-a160-ec3a3507e2e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:07.115665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4236f9-0cf6-4ba9-a160-ec3a3507e2e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:07.115940 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4236f9-0cf6-4ba9-a160-ec3a3507e2e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:07.116208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4236f9-0cf6-4ba9-a160-ec3a3507e2e4 admin admin] PATH_INFO: `/v3/projects/e367eb231b38468c8bd6ebe483d6ecfb/users/429548cbe4544ea2b2632bf17bb4bd5a/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:07.131507 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4236f9-0cf6-4ba9-a160-ec3a3507e2e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e367eb231b38468c8bd6ebe483d6ecfb, user_id=429548cbe4544ea2b2632bf17bb4bd5a, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:07.134143 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e4236f9-0cf6-4ba9-a160-ec3a3507e2e4 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:07.292127 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:07.301252 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1027/2017] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:07 2022] PUT /identity/v3/projects/e367eb231b38468c8bd6ebe483d6ecfb/users/429548cbe4544ea2b2632bf17bb4bd5a/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:07.302699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4236f9-0cf6-4ba9-a160-ec3a3507e2e4 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:07.307166 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 991/2018] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:06 2022] POST /identity/v3/users => generated 283 bytes in 391 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:07.307300 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-0e4236f9-0cf6-4ba9-a160-ec3a3507e2e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:07.308213 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b039b7-eeb4-4072-8e4a-4acc803ec508 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:07.330808 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b039b7-eeb4-4072-8e4a-4acc803ec508 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:07.396224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4236f9-0cf6-4ba9-a160-ec3a3507e2e4 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:07.399082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ac58cf-7aec-4c70-8c62-b0d5df8b19d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:07.399771 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ac58cf-7aec-4c70-8c62-b0d5df8b19d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:07.400103 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ac58cf-7aec-4c70-8c62-b0d5df8b19d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:07.400376 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ac58cf-7aec-4c70-8c62-b0d5df8b19d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:07.401049 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ac58cf-7aec-4c70-8c62-b0d5df8b19d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:07.401514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07cb892-5204-4f6c-9e65-817b09814972 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:07.402092 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07cb892-5204-4f6c-9e65-817b09814972 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:07.402410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07cb892-5204-4f6c-9e65-817b09814972 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:07.402669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ac58cf-7aec-4c70-8c62-b0d5df8b19d0 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:07.402783 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07cb892-5204-4f6c-9e65-817b09814972 admin admin] PATH_INFO: `/v3/projects/4e5c2dfbf6e8451aa06c3193fc850a22` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:07.403883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07cb892-5204-4f6c-9e65-817b09814972 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4e5c2dfbf6e8451aa06c3193fc850a22)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:07.405804 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07cb892-5204-4f6c-9e65-817b09814972 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:07.413229 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 992/2019] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:07 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:03:07.414294 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52fdbb1-65b2-47e6-9b5b-461e4260b4c4 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:07.414607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52fdbb1-65b2-47e6-9b5b-461e4260b4c4 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:07.414884 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d52fdbb1-65b2-47e6-9b5b-461e4260b4c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:07.443467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-d52fdbb1-65b2-47e6-9b5b-461e4260b4c4 None None] MFA Rules not processed for user `429548cbe4544ea2b2632bf17bb4bd5a`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:07.444694 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d52fdbb1-65b2-47e6-9b5b-461e4260b4c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:07.511222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 993/2020] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:07 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:07.512095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52fdbb1-65b2-47e6-9b5b-461e4260b4c4 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:07.513483 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d52fdbb1-65b2-47e6-9b5b-461e4260b4c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:07.526694 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e07cb892-5204-4f6c-9e65-817b09814972 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4e5c2dfbf6e8451aa06c3193fc850a22', 'request_id': 'req-e07cb892-5204-4f6c-9e65-817b09814972'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:07.527452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d52fdbb1-65b2-47e6-9b5b-461e4260b4c4 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:07.588081 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae8aa11-f715-47e4-9497-dceb98a210f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:07.588717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae8aa11-f715-47e4-9497-dceb98a210f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:07.589001 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae8aa11-f715-47e4-9497-dceb98a210f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:07.589273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae8aa11-f715-47e4-9497-dceb98a210f7 admin admin] PATH_INFO: `/v3/projects/df1ff703fc014f5fb2e1f918ae48ea08` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:07.590457 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae8aa11-f715-47e4-9497-dceb98a210f7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=df1ff703fc014f5fb2e1f918ae48ea08)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:07.592382 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae8aa11-f715-47e4-9497-dceb98a210f7 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:07.616749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e07cb892-5204-4f6c-9e65-817b09814972 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4e5c2dfbf6e8451aa06c3193fc850a22', 'request_id': 'req-e07cb892-5204-4f6c-9e65-817b09814972'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:07.623356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e07cb892-5204-4f6c-9e65-817b09814972 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4e5c2dfbf6e8451aa06c3193fc850a22', 'request_id': 'req-e07cb892-5204-4f6c-9e65-817b09814972'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:07.634551 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e07cb892-5204-4f6c-9e65-817b09814972 admin admin] The token cache is being invalidate because project 4e5c2dfbf6e8451aa06c3193fc850a22 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:07.634839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e07cb892-5204-4f6c-9e65-817b09814972 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:07.635495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e07cb892-5204-4f6c-9e65-817b09814972 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:07.636980 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1028/2021] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:07 2022] DELETE /identity/v3/projects/4e5c2dfbf6e8451aa06c3193fc850a22 => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:07.637771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07cb892-5204-4f6c-9e65-817b09814972 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:07.639053 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-e07cb892-5204-4f6c-9e65-817b09814972 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:07.651141 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07cb892-5204-4f6c-9e65-817b09814972 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:07.655674 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354037b1-339b-4c9c-87a8-c6520a3976c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:07.656247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354037b1-339b-4c9c-87a8-c6520a3976c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:07.656529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354037b1-339b-4c9c-87a8-c6520a3976c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:07.656794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354037b1-339b-4c9c-87a8-c6520a3976c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:07.657420 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354037b1-339b-4c9c-87a8-c6520a3976c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:07.659628 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354037b1-339b-4c9c-87a8-c6520a3976c2 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:07.669139 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1029/2022] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:07 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:07.669878 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354037b1-339b-4c9c-87a8-c6520a3976c2 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:07.671486 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-354037b1-339b-4c9c-87a8-c6520a3976c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:07.684847 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354037b1-339b-4c9c-87a8-c6520a3976c2 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:07.689515 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68e04a5-84d7-4915-ac49-d4890925505e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:07.692341 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68e04a5-84d7-4915-ac49-d4890925505e admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:07.692341 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68e04a5-84d7-4915-ac49-d4890925505e admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:07.692341 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68e04a5-84d7-4915-ac49-d4890925505e admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:07.692341 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68e04a5-84d7-4915-ac49-d4890925505e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:07.693037 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68e04a5-84d7-4915-ac49-d4890925505e admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:07.705411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-aae8aa11-f715-47e4-9497-dceb98a210f7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'df1ff703fc014f5fb2e1f918ae48ea08', 'request_id': 'req-aae8aa11-f715-47e4-9497-dceb98a210f7'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:07.810762 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-aae8aa11-f715-47e4-9497-dceb98a210f7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'df1ff703fc014f5fb2e1f918ae48ea08', 'request_id': 'req-aae8aa11-f715-47e4-9497-dceb98a210f7'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:07.817594 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-aae8aa11-f715-47e4-9497-dceb98a210f7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'df1ff703fc014f5fb2e1f918ae48ea08', 'request_id': 'req-aae8aa11-f715-47e4-9497-dceb98a210f7'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:07.820204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1030/2023] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:07 2022] POST /identity/v3/projects => generated 407 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:07.821166 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68e04a5-84d7-4915-ac49-d4890925505e admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:07.822919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f68e04a5-84d7-4915-ac49-d4890925505e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:07.833497 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-aae8aa11-f715-47e4-9497-dceb98a210f7 admin admin] The token cache is being invalidate because project df1ff703fc014f5fb2e1f918ae48ea08 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:07.833833 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-aae8aa11-f715-47e4-9497-dceb98a210f7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:07.834525 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-aae8aa11-f715-47e4-9497-dceb98a210f7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:07.842084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 994/2024] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:07 2022] DELETE /identity/v3/projects/df1ff703fc014f5fb2e1f918ae48ea08 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:07.842084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae8aa11-f715-47e4-9497-dceb98a210f7 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:07.842084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-aae8aa11-f715-47e4-9497-dceb98a210f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:07.896874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68e04a5-84d7-4915-ac49-d4890925505e admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:07.904032 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31e40c7-c2bb-4fe6-944f-efb091088023 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:07.904955 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31e40c7-c2bb-4fe6-944f-efb091088023 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:07.905140 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31e40c7-c2bb-4fe6-944f-efb091088023 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:07.905503 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31e40c7-c2bb-4fe6-944f-efb091088023 admin admin] PATH_INFO: `/v3/users/b5add60179a6493e9f58a42719fcc95a` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:07.906926 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31e40c7-c2bb-4fe6-944f-efb091088023 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b5add60179a6493e9f58a42719fcc95a)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:07.909606 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31e40c7-c2bb-4fe6-944f-efb091088023 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:07.915273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae8aa11-f715-47e4-9497-dceb98a210f7 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:07.920202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc17b6c6-77bc-4ea2-ac8c-7042e72325b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:07.920875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc17b6c6-77bc-4ea2-ac8c-7042e72325b4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:07.921166 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc17b6c6-77bc-4ea2-ac8c-7042e72325b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:07.921428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc17b6c6-77bc-4ea2-ac8c-7042e72325b4 admin admin] PATH_INFO: `/v3/projects/ca45b248032f449da726c9dd9bba62a0/users/5fd44db8498f47c2910a623fe7fb6f26/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:07.937133 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc17b6c6-77bc-4ea2-ac8c-7042e72325b4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ca45b248032f449da726c9dd9bba62a0, user_id=5fd44db8498f47c2910a623fe7fb6f26, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:07.940308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc17b6c6-77bc-4ea2-ac8c-7042e72325b4 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:07.976699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:08.045031 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 995/2025] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:07 2022] PUT /identity/v3/projects/ca45b248032f449da726c9dd9bba62a0/users/5fd44db8498f47c2910a623fe7fb6f26/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:08.045848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc17b6c6-77bc-4ea2-ac8c-7042e72325b4 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:08.047457 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-fc17b6c6-77bc-4ea2-ac8c-7042e72325b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:08.115053 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc17b6c6-77bc-4ea2-ac8c-7042e72325b4 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:08.119942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2397ef69-cd1e-4af5-924a-1319543da98c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:08.120552 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2397ef69-cd1e-4af5-924a-1319543da98c admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:08.120830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2397ef69-cd1e-4af5-924a-1319543da98c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:08.121153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2397ef69-cd1e-4af5-924a-1319543da98c admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:08.121785 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2397ef69-cd1e-4af5-924a-1319543da98c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:08.123772 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2397ef69-cd1e-4af5-924a-1319543da98c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:08.125724 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f31e40c7-c2bb-4fe6-944f-efb091088023 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b5add60179a6493e9f58a42719fcc95a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:08.210593 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f31e40c7-c2bb-4fe6-944f-efb091088023 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b5add60179a6493e9f58a42719fcc95a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:08.223489 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f31e40c7-c2bb-4fe6-944f-efb091088023 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b5add60179a6493e9f58a42719fcc95a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:08.224178 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f31e40c7-c2bb-4fe6-944f-efb091088023 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b5add60179a6493e9f58a42719fcc95a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:08.231104 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f31e40c7-c2bb-4fe6-944f-efb091088023 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b5add60179a6493e9f58a42719fcc95a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:08.233759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 996/2026] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:08 2022] POST /identity/v3/users => generated 310 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:08.233759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cc362b-b060-4dc0-b349-5e5554b57326 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:08.233759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cc362b-b060-4dc0-b349-5e5554b57326 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:08.233759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cc362b-b060-4dc0-b349-5e5554b57326 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:08.237321 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1031/2027] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:07 2022] DELETE /identity/v3/users/b5add60179a6493e9f58a42719fcc95a => generated 0 bytes in 417 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:08.238145 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31e40c7-c2bb-4fe6-944f-efb091088023 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:08.239649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f31e40c7-c2bb-4fe6-944f-efb091088023 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:08.259054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-c7cc362b-b060-4dc0-b349-5e5554b57326 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:08.260401 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-c7cc362b-b060-4dc0-b349-5e5554b57326 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:08.308359 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31e40c7-c2bb-4fe6-944f-efb091088023 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:08.313998 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df683884-a286-4095-9d03-5355145083cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:08.314941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df683884-a286-4095-9d03-5355145083cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:08.315268 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df683884-a286-4095-9d03-5355145083cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:08.315542 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df683884-a286-4095-9d03-5355145083cd admin admin] PATH_INFO: `/v3/users/fe0780d583af4579954b393f0798cce7` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:08.316705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df683884-a286-4095-9d03-5355145083cd admin admin] RBAC: Authorizing `identity:delete_user(user_id=fe0780d583af4579954b393f0798cce7)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:08.318659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df683884-a286-4095-9d03-5355145083cd admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:08.327921 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 997/2028] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:08.329427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b0e5a6-2eed-45c4-8680-5d5454197d59 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:08.330506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b0e5a6-2eed-45c4-8680-5d5454197d59 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:08.331033 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b0e5a6-2eed-45c4-8680-5d5454197d59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:08.358773 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-c2b0e5a6-2eed-45c4-8680-5d5454197d59 None None] MFA Rules not processed for user `5fd44db8498f47c2910a623fe7fb6f26`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:08.359978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-c2b0e5a6-2eed-45c4-8680-5d5454197d59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:08.422909 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 998/2029] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:08 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:08.423911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b0e5a6-2eed-45c4-8680-5d5454197d59 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:08.425239 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-c2b0e5a6-2eed-45c4-8680-5d5454197d59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:08.438152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b0e5a6-2eed-45c4-8680-5d5454197d59 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:08.443607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ecd2e2-2c12-4e21-aa31-c39538bb0855 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:08.444183 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ecd2e2-2c12-4e21-aa31-c39538bb0855 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:08.444462 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ecd2e2-2c12-4e21-aa31-c39538bb0855 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:08.444732 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ecd2e2-2c12-4e21-aa31-c39538bb0855 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:08.445368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ecd2e2-2c12-4e21-aa31-c39538bb0855 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:08.446974 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ecd2e2-2c12-4e21-aa31-c39538bb0855 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:08.457282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 999/2030] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:08 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:08.458721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc784555-715e-4937-baf6-c25886f0b392 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:08.459307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc784555-715e-4937-baf6-c25886f0b392 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:08.460001 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc784555-715e-4937-baf6-c25886f0b392 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:08.479012 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-bc784555-715e-4937-baf6-c25886f0b392 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:08.480215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-bc784555-715e-4937-baf6-c25886f0b392 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:08.490997 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1000/2031] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:08.491873 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc784555-715e-4937-baf6-c25886f0b392 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:08.493193 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-bc784555-715e-4937-baf6-c25886f0b392 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:08.505494 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc784555-715e-4937-baf6-c25886f0b392 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:08.510397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3ff45e-f234-427e-b880-98fc22c6d32e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:08.511275 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3ff45e-f234-427e-b880-98fc22c6d32e admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:08.511653 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3ff45e-f234-427e-b880-98fc22c6d32e admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:08.511962 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3ff45e-f234-427e-b880-98fc22c6d32e admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:08.512582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3ff45e-f234-427e-b880-98fc22c6d32e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:08.514490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3ff45e-f234-427e-b880-98fc22c6d32e admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:08.568434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-df683884-a286-4095-9d03-5355145083cd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fe0780d583af4579954b393f0798cce7'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:08.649679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1001/2032] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:08 2022] POST /identity/v3/projects => generated 367 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:08.649679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3ff45e-f234-427e-b880-98fc22c6d32e admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:08.679203 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-df683884-a286-4095-9d03-5355145083cd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fe0780d583af4579954b393f0798cce7'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:08.692608 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-df683884-a286-4095-9d03-5355145083cd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fe0780d583af4579954b393f0798cce7'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:08.693272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-df683884-a286-4095-9d03-5355145083cd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fe0780d583af4579954b393f0798cce7'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:08.700095 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-df683884-a286-4095-9d03-5355145083cd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fe0780d583af4579954b393f0798cce7'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:08.706061 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1032/2033] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:08 2022] DELETE /identity/v3/users/fe0780d583af4579954b393f0798cce7 => generated 0 bytes in 468 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:08.706892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df683884-a286-4095-9d03-5355145083cd admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:08.708219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-df683884-a286-4095-9d03-5355145083cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:08.712016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3ff45e-f234-427e-b880-98fc22c6d32e admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:08.776273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7235e7cc-f60c-4023-a463-fb9c69147194 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:08.777019 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7235e7cc-f60c-4023-a463-fb9c69147194 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:08.777302 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7235e7cc-f60c-4023-a463-fb9c69147194 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:08.777578 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7235e7cc-f60c-4023-a463-fb9c69147194 admin admin] PATH_INFO: `/v3/projects/0663e86bf6e048d4bee2938e9ee39906/users/ceddaa716eb94e2cbcca0eaa1d6096fc/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:08.779075 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df683884-a286-4095-9d03-5355145083cd admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:08.784099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8b8496-8067-4f36-b78f-3afe7adf259a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:08.784932 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8b8496-8067-4f36-b78f-3afe7adf259a admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:08.789017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8b8496-8067-4f36-b78f-3afe7adf259a admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:08.789017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8b8496-8067-4f36-b78f-3afe7adf259a admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:08.789017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8b8496-8067-4f36-b78f-3afe7adf259a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:03:08.789017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8b8496-8067-4f36-b78f-3afe7adf259a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:08.789017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d8b8496-8067-4f36-b78f-3afe7adf259a admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:08.795512 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7235e7cc-f60c-4023-a463-fb9c69147194 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0663e86bf6e048d4bee2938e9ee39906, user_id=ceddaa716eb94e2cbcca0eaa1d6096fc, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:08.798276 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7235e7cc-f60c-4023-a463-fb9c69147194 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:08.805283 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1033/2034] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:03:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:03:08.809821 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8b8496-8067-4f36-b78f-3afe7adf259a admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:08.809821 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8d8b8496-8067-4f36-b78f-3afe7adf259a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:08.820952 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8b8496-8067-4f36-b78f-3afe7adf259a admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:08.825466 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf76654f-8c2c-4048-a9cd-a136ef8f2e80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:08.826044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf76654f-8c2c-4048-a9cd-a136ef8f2e80 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:08.826322 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf76654f-8c2c-4048-a9cd-a136ef8f2e80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:08.826619 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf76654f-8c2c-4048-a9cd-a136ef8f2e80 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:08.827232 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf76654f-8c2c-4048-a9cd-a136ef8f2e80 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:08.829150 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf76654f-8c2c-4048-a9cd-a136ef8f2e80 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:08.891632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1002/2035] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:08 2022] PUT /identity/v3/projects/0663e86bf6e048d4bee2938e9ee39906/users/ceddaa716eb94e2cbcca0eaa1d6096fc/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:08.893200 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d18836-3566-4899-b8b9-277bba39e504 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:08.893515 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d18836-3566-4899-b8b9-277bba39e504 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:08.893787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d18836-3566-4899-b8b9-277bba39e504 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:08.913251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-79d18836-3566-4899-b8b9-277bba39e504 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:08.914623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-79d18836-3566-4899-b8b9-277bba39e504 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:08.914946 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1034/2036] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:08 2022] POST /identity/v3/users => generated 290 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:08.915825 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf76654f-8c2c-4048-a9cd-a136ef8f2e80 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:08.917510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-cf76654f-8c2c-4048-a9cd-a136ef8f2e80 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:08.934671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf76654f-8c2c-4048-a9cd-a136ef8f2e80 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:08.978365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1003/2037] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:08.979205 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d18836-3566-4899-b8b9-277bba39e504 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:08.988957 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:08.992781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d18836-3566-4899-b8b9-277bba39e504 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:08.997859 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf5acb6-51c3-4fa4-aad1-037f0f2d8616 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:08.998645 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf5acb6-51c3-4fa4-aad1-037f0f2d8616 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:08.999025 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf5acb6-51c3-4fa4-aad1-037f0f2d8616 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:08.999443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf5acb6-51c3-4fa4-aad1-037f0f2d8616 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:09.000225 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf5acb6-51c3-4fa4-aad1-037f0f2d8616 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:09.002394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf5acb6-51c3-4fa4-aad1-037f0f2d8616 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:09.004392 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ff10a3-271f-4405-b88c-f821ef0b4f4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:09.005103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ff10a3-271f-4405-b88c-f821ef0b4f4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:09.005491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ff10a3-271f-4405-b88c-f821ef0b4f4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:09.005863 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ff10a3-271f-4405-b88c-f821ef0b4f4a admin admin] PATH_INFO: `/v3/projects/fce3614cf145410da0f9185326de2813` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:09.007253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ff10a3-271f-4405-b88c-f821ef0b4f4a admin admin] RBAC: Authorizing `identity:delete_project(project_id=fce3614cf145410da0f9185326de2813)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:09.009358 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ff10a3-271f-4405-b88c-f821ef0b4f4a admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:09.019856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:09.174367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1004/2038] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:08 2022] POST /identity/v3/projects => generated 385 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:09.174367 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf5acb6-51c3-4fa4-aad1-037f0f2d8616 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:09.178078 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-6bf5acb6-51c3-4fa4-aad1-037f0f2d8616 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:09.223767 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e2ff10a3-271f-4405-b88c-f821ef0b4f4a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fce3614cf145410da0f9185326de2813', 'request_id': 'req-e2ff10a3-271f-4405-b88c-f821ef0b4f4a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:09.266918 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf5acb6-51c3-4fa4-aad1-037f0f2d8616 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:09.324668 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e2ff10a3-271f-4405-b88c-f821ef0b4f4a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fce3614cf145410da0f9185326de2813', 'request_id': 'req-e2ff10a3-271f-4405-b88c-f821ef0b4f4a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:09.331545 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e2ff10a3-271f-4405-b88c-f821ef0b4f4a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fce3614cf145410da0f9185326de2813', 'request_id': 'req-e2ff10a3-271f-4405-b88c-f821ef0b4f4a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:09.342028 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f557edc-c9a1-4fb4-8604-ddc11743420d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:09.342706 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f557edc-c9a1-4fb4-8604-ddc11743420d admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:09.342982 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f557edc-c9a1-4fb4-8604-ddc11743420d admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:09.343250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f557edc-c9a1-4fb4-8604-ddc11743420d admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:09.343917 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f557edc-c9a1-4fb4-8604-ddc11743420d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:09.346368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f557edc-c9a1-4fb4-8604-ddc11743420d admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:09.354647 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e2ff10a3-271f-4405-b88c-f821ef0b4f4a admin admin] The token cache is being invalidate because project fce3614cf145410da0f9185326de2813 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:09.355012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e2ff10a3-271f-4405-b88c-f821ef0b4f4a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:09.355739 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e2ff10a3-271f-4405-b88c-f821ef0b4f4a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:09.360013 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1005/2039] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:09 2022] GET /identity/v3/roles => generated 1420 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:09.364027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1035/2040] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:08 2022] DELETE /identity/v3/projects/fce3614cf145410da0f9185326de2813 => generated 0 bytes in 442 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:09.364138 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f557edc-c9a1-4fb4-8604-ddc11743420d admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:09.364323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ff10a3-271f-4405-b88c-f821ef0b4f4a admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:09.364323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-e2ff10a3-271f-4405-b88c-f821ef0b4f4a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:09.364493 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2f557edc-c9a1-4fb4-8604-ddc11743420d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:09.373701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f557edc-c9a1-4fb4-8604-ddc11743420d admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:09.378372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c786a9a-12ac-42b0-ac7c-f41c6cebcff9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:09.378971 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c786a9a-12ac-42b0-ac7c-f41c6cebcff9 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:09.379246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c786a9a-12ac-42b0-ac7c-f41c6cebcff9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:09.379509 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c786a9a-12ac-42b0-ac7c-f41c6cebcff9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:09.380214 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c786a9a-12ac-42b0-ac7c-f41c6cebcff9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:09.381884 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c786a9a-12ac-42b0-ac7c-f41c6cebcff9 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:09.392949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1006/2041] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:09 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:09.393649 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c786a9a-12ac-42b0-ac7c-f41c6cebcff9 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:09.395306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3c786a9a-12ac-42b0-ac7c-f41c6cebcff9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:09.407526 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c786a9a-12ac-42b0-ac7c-f41c6cebcff9 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:09.413080 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a1e486-a487-42d1-b4e7-4de21628d3f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:09.413727 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a1e486-a487-42d1-b4e7-4de21628d3f6 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:09.414008 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a1e486-a487-42d1-b4e7-4de21628d3f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:09.414271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a1e486-a487-42d1-b4e7-4de21628d3f6 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:09.414936 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a1e486-a487-42d1-b4e7-4de21628d3f6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:09.419889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a1e486-a487-42d1-b4e7-4de21628d3f6 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:09.420024 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ff10a3-271f-4405-b88c-f821ef0b4f4a admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:09.421012 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:09.421747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd536db-a723-4278-a35c-5f27333079b7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:09.422356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd536db-a723-4278-a35c-5f27333079b7 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:09.422645 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd536db-a723-4278-a35c-5f27333079b7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:09.422911 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd536db-a723-4278-a35c-5f27333079b7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:09.424919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-dcd536db-a723-4278-a35c-5f27333079b7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:09.437118 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd536db-a723-4278-a35c-5f27333079b7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:09.438571 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcd536db-a723-4278-a35c-5f27333079b7 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:09.448344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1036/2042] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:09.449094 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd536db-a723-4278-a35c-5f27333079b7 service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:09.450322 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-dcd536db-a723-4278-a35c-5f27333079b7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:09.461874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd536db-a723-4278-a35c-5f27333079b7 service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:09.466920 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af714ec7-d1d8-4c26-ad64-1670334987bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:09.467596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af714ec7-d1d8-4c26-ad64-1670334987bf admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:09.468027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af714ec7-d1d8-4c26-ad64-1670334987bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:09.468381 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af714ec7-d1d8-4c26-ad64-1670334987bf admin admin] PATH_INFO: `/v3/projects/0663e86bf6e048d4bee2938e9ee39906/users/ceddaa716eb94e2cbcca0eaa1d6096fc/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:09.471380 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af714ec7-d1d8-4c26-ad64-1670334987bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=0663e86bf6e048d4bee2938e9ee39906, user_id=ceddaa716eb94e2cbcca0eaa1d6096fc, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:09.474074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af714ec7-d1d8-4c26-ad64-1670334987bf admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:09.484893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1037/2043] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:09 2022] PUT /identity/v3/projects/0663e86bf6e048d4bee2938e9ee39906/users/ceddaa716eb94e2cbcca0eaa1d6096fc/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:09.486203 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fefeab3-5a2a-4190-aeef-256f78e181b1 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:09.486618 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fefeab3-5a2a-4190-aeef-256f78e181b1 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:09.486900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fefeab3-5a2a-4190-aeef-256f78e181b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:09.506606 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1007/2044] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:09 2022] POST /identity/v3/users => generated 293 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:09.507799 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a1e486-a487-42d1-b4e7-4de21628d3f6 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:09.509387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-1fefeab3-5a2a-4190-aeef-256f78e181b1 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:09.511121 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-1fefeab3-5a2a-4190-aeef-256f78e181b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:09.518209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a1e486-a487-42d1-b4e7-4de21628d3f6 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:09.576520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1038/2045] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:09.577991 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dbf636-2997-4c7a-a67f-475663d112af None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:09.578448 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dbf636-2997-4c7a-a67f-475663d112af None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:09.578722 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99dbf636-2997-4c7a-a67f-475663d112af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:09.586579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2093d9ab-32b9-415c-99f6-e2a4a6a35172 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:09.587829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2093d9ab-32b9-415c-99f6-e2a4a6a35172 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:09.587829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2093d9ab-32b9-415c-99f6-e2a4a6a35172 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:09.587829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2093d9ab-32b9-415c-99f6-e2a4a6a35172 admin admin] PATH_INFO: `/v3/projects/53ec96afe7fa43c79713342da619af04/users/ab51b4b0402c4e07b9e8459b8479d25a/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:09.603060 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2093d9ab-32b9-415c-99f6-e2a4a6a35172 admin admin] RBAC: Authorizing `identity:create_grant(project_id=53ec96afe7fa43c79713342da619af04, user_id=ab51b4b0402c4e07b9e8459b8479d25a, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:09.605701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2093d9ab-32b9-415c-99f6-e2a4a6a35172 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:09.609055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-99dbf636-2997-4c7a-a67f-475663d112af None None] MFA Rules not processed for user `ceddaa716eb94e2cbcca0eaa1d6096fc`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:09.610262 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-99dbf636-2997-4c7a-a67f-475663d112af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:09.663572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1008/2046] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:09 2022] PUT /identity/v3/projects/53ec96afe7fa43c79713342da619af04/users/ab51b4b0402c4e07b9e8459b8479d25a/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:09.667290 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2093d9ab-32b9-415c-99f6-e2a4a6a35172 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:09.690537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1039/2047] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:09 2022] POST /identity/v3/auth/tokens => generated 3351 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:09.690537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7210aa96-57f2-4c14-8630-05f18c446678 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:09.690537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7210aa96-57f2-4c14-8630-05f18c446678 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:09.694746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7210aa96-57f2-4c14-8630-05f18c446678 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:09.708165 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:09.721820 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-7210aa96-57f2-4c14-8630-05f18c446678 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:09.723094 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-7210aa96-57f2-4c14-8630-05f18c446678 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:09.748180 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2093d9ab-32b9-415c-99f6-e2a4a6a35172 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:09.753931 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e8c9fc-8506-4402-aa17-e9842c14ebe2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:09.754714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e8c9fc-8506-4402-aa17-e9842c14ebe2 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:09.754997 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e8c9fc-8506-4402-aa17-e9842c14ebe2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:09.755272 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e8c9fc-8506-4402-aa17-e9842c14ebe2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:09.755925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e8c9fc-8506-4402-aa17-e9842c14ebe2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:09.757667 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e8c9fc-8506-4402-aa17-e9842c14ebe2 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:09.769088 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1009/2048] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:09 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:09.769929 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e8c9fc-8506-4402-aa17-e9842c14ebe2 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:09.780398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e8c9fc-8506-4402-aa17-e9842c14ebe2 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:09.785175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fdbbd2-ec9d-498d-a7cd-d6b02e8d9c60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:09.785782 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fdbbd2-ec9d-498d-a7cd-d6b02e8d9c60 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:09.786064 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fdbbd2-ec9d-498d-a7cd-d6b02e8d9c60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:09.786363 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19fdbbd2-ec9d-498d-a7cd-d6b02e8d9c60 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:09.787008 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fdbbd2-ec9d-498d-a7cd-d6b02e8d9c60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:09.788589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19fdbbd2-ec9d-498d-a7cd-d6b02e8d9c60 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:09.792846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1040/2049] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:09.793815 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7210aa96-57f2-4c14-8630-05f18c446678 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:09.795441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-7210aa96-57f2-4c14-8630-05f18c446678 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:09.803977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1010/2050] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:09 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:09.805123 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fdbbd2-ec9d-498d-a7cd-d6b02e8d9c60 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:09.817867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19fdbbd2-ec9d-498d-a7cd-d6b02e8d9c60 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:09.822867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e4db2a-6f94-4dec-a3a6-72cb60705535 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:09.823513 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e4db2a-6f94-4dec-a3a6-72cb60705535 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:09.823798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e4db2a-6f94-4dec-a3a6-72cb60705535 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:09.824075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e4db2a-6f94-4dec-a3a6-72cb60705535 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:09.824718 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e4db2a-6f94-4dec-a3a6-72cb60705535 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:09.826343 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e4db2a-6f94-4dec-a3a6-72cb60705535 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:09.838643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1011/2051] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:09 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:09.839493 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e4db2a-6f94-4dec-a3a6-72cb60705535 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:09.849398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e4db2a-6f94-4dec-a3a6-72cb60705535 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:09.854063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ab8296-9e9a-4b28-8c17-35c22bddeb16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:09.854677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ab8296-9e9a-4b28-8c17-35c22bddeb16 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:09.854958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ab8296-9e9a-4b28-8c17-35c22bddeb16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:09.855224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ab8296-9e9a-4b28-8c17-35c22bddeb16 admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:09.855852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ab8296-9e9a-4b28-8c17-35c22bddeb16 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:03:09.856151 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ab8296-9e9a-4b28-8c17-35c22bddeb16 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:09.857713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ab8296-9e9a-4b28-8c17-35c22bddeb16 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:09.859491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7210aa96-57f2-4c14-8630-05f18c446678 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:09.863969 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc33ae2-0877-477a-8c26-e8007049f488 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:09.864572 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc33ae2-0877-477a-8c26-e8007049f488 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:09.864849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc33ae2-0877-477a-8c26-e8007049f488 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:09.865119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc33ae2-0877-477a-8c26-e8007049f488 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:09.870602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1012/2052] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:03:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:03:09.871389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ab8296-9e9a-4b28-8c17-35c22bddeb16 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:09.881069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ab8296-9e9a-4b28-8c17-35c22bddeb16 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:09.885769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efcfcae-2814-42ab-87bc-cabaf1767090 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:09.886432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efcfcae-2814-42ab-87bc-cabaf1767090 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:09.886713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efcfcae-2814-42ab-87bc-cabaf1767090 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:09.886985 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efcfcae-2814-42ab-87bc-cabaf1767090 admin admin] PATH_INFO: `/v3/projects/53ec96afe7fa43c79713342da619af04/users/ab51b4b0402c4e07b9e8459b8479d25a/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:09.888865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efcfcae-2814-42ab-87bc-cabaf1767090 admin admin] RBAC: Authorizing `identity:create_grant(project_id=53ec96afe7fa43c79713342da619af04, user_id=ab51b4b0402c4e07b9e8459b8479d25a, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:09.891518 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efcfcae-2814-42ab-87bc-cabaf1767090 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:09.902657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1013/2053] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:09 2022] PUT /identity/v3/projects/53ec96afe7fa43c79713342da619af04/users/ab51b4b0402c4e07b9e8459b8479d25a/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:09.903661 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efcfcae-2814-42ab-87bc-cabaf1767090 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:09.938492 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc33ae2-0877-477a-8c26-e8007049f488 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:09.940626 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc33ae2-0877-477a-8c26-e8007049f488 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:09.950862 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1041/2054] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:09 2022] GET /identity/v3/auth/tokens => generated 3351 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:09.951770 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc33ae2-0877-477a-8c26-e8007049f488 service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:09.961874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc33ae2-0877-477a-8c26-e8007049f488 service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:09.981448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efcfcae-2814-42ab-87bc-cabaf1767090 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:09.986960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3540058c-bd4d-4060-8619-7e2fbbe47cb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:09.987748 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3540058c-bd4d-4060-8619-7e2fbbe47cb5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:09.988143 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3540058c-bd4d-4060-8619-7e2fbbe47cb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:09.988418 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3540058c-bd4d-4060-8619-7e2fbbe47cb5 admin admin] PATH_INFO: `/v3/projects/3c0e6b9e809c43a8975c25124b157c6c/users/ecdf2746343945749f54839322419dbe/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:10.006114 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3540058c-bd4d-4060-8619-7e2fbbe47cb5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3c0e6b9e809c43a8975c25124b157c6c, user_id=ecdf2746343945749f54839322419dbe, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:10.009344 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3540058c-bd4d-4060-8619-7e2fbbe47cb5 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:10.013429 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:10.037866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61dfa9bc-79af-43e1-8cf1-c8ae48b3186a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:10.038527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dfa9bc-79af-43e1-8cf1-c8ae48b3186a admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:10.038803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dfa9bc-79af-43e1-8cf1-c8ae48b3186a admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:10.039069 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dfa9bc-79af-43e1-8cf1-c8ae48b3186a admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:10.039701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61dfa9bc-79af-43e1-8cf1-c8ae48b3186a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:10.041627 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61dfa9bc-79af-43e1-8cf1-c8ae48b3186a admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:10.090744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:10.097755 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1014/2055] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:09 2022] PUT /identity/v3/projects/3c0e6b9e809c43a8975c25124b157c6c/users/ecdf2746343945749f54839322419dbe/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:10.099094 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4270483-76dd-4187-92f6-f80798290abd None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:10.099373 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4270483-76dd-4187-92f6-f80798290abd None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:10.099642 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4270483-76dd-4187-92f6-f80798290abd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:10.126394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-c4270483-76dd-4187-92f6-f80798290abd None None] MFA Rules not processed for user `ab51b4b0402c4e07b9e8459b8479d25a`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:10.127756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-c4270483-76dd-4187-92f6-f80798290abd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:10.159835 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1042/2056] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:09 2022] POST /identity/v3/projects => generated 385 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:10.159835 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61dfa9bc-79af-43e1-8cf1-c8ae48b3186a admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:10.161490 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-61dfa9bc-79af-43e1-8cf1-c8ae48b3186a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:10.196306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1015/2057] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:10 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:10.197572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8726be-3e4d-43ca-9583-8ee4ed4c7c94 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:10.197867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8726be-3e4d-43ca-9583-8ee4ed4c7c94 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:10.198141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8726be-3e4d-43ca-9583-8ee4ed4c7c94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:10.228977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-7e8726be-3e4d-43ca-9583-8ee4ed4c7c94 None None] MFA Rules not processed for user `ecdf2746343945749f54839322419dbe`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:10.230189 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-7e8726be-3e4d-43ca-9583-8ee4ed4c7c94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:10.241992 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61dfa9bc-79af-43e1-8cf1-c8ae48b3186a admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:10.246807 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e1c24e-2706-4bba-8e50-5bc216031f53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:10.247478 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e1c24e-2706-4bba-8e50-5bc216031f53 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:10.247756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e1c24e-2706-4bba-8e50-5bc216031f53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:10.248031 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e1c24e-2706-4bba-8e50-5bc216031f53 admin admin] PATH_INFO: `/v3/projects/ad95901102c9425583cf14ee0698d656` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:10.249165 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e1c24e-2706-4bba-8e50-5bc216031f53 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ad95901102c9425583cf14ee0698d656)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:10.251144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e1c24e-2706-4bba-8e50-5bc216031f53 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:10.295604 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1016/2058] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:10 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:10.296579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8726be-3e4d-43ca-9583-8ee4ed4c7c94 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:10.307632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8726be-3e4d-43ca-9583-8ee4ed4c7c94 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:10.312535 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244a91ad-0d9f-4370-ad78-276a367580dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:10.313214 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244a91ad-0d9f-4370-ad78-276a367580dd admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:10.313489 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244a91ad-0d9f-4370-ad78-276a367580dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:10.313760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244a91ad-0d9f-4370-ad78-276a367580dd admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:10.314961 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244a91ad-0d9f-4370-ad78-276a367580dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:10.316899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244a91ad-0d9f-4370-ad78-276a367580dd admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:10.351483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-00e1c24e-2706-4bba-8e50-5bc216031f53 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ad95901102c9425583cf14ee0698d656', 'request_id': 'req-00e1c24e-2706-4bba-8e50-5bc216031f53'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:10.416167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-00e1c24e-2706-4bba-8e50-5bc216031f53 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ad95901102c9425583cf14ee0698d656', 'request_id': 'req-00e1c24e-2706-4bba-8e50-5bc216031f53'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:10.425762 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1017/2059] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:10 2022] POST /identity/v3/users => generated 293 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:10.425934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-00e1c24e-2706-4bba-8e50-5bc216031f53 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ad95901102c9425583cf14ee0698d656', 'request_id': 'req-00e1c24e-2706-4bba-8e50-5bc216031f53'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:10.426048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244a91ad-0d9f-4370-ad78-276a367580dd admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:10.440569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-00e1c24e-2706-4bba-8e50-5bc216031f53 admin admin] The token cache is being invalidate because project ad95901102c9425583cf14ee0698d656 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:10.440862 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-00e1c24e-2706-4bba-8e50-5bc216031f53 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:10.441535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-00e1c24e-2706-4bba-8e50-5bc216031f53 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:10.443223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1043/2060] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:10 2022] DELETE /identity/v3/projects/ad95901102c9425583cf14ee0698d656 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:10.444015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e1c24e-2706-4bba-8e50-5bc216031f53 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:10.445357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-00e1c24e-2706-4bba-8e50-5bc216031f53 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:10.501995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244a91ad-0d9f-4370-ad78-276a367580dd admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:10.505966 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a571be2-5ad0-49c1-8975-29a00a92d16e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:10.506963 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a571be2-5ad0-49c1-8975-29a00a92d16e service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:10.506963 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a571be2-5ad0-49c1-8975-29a00a92d16e service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:10.507159 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a571be2-5ad0-49c1-8975-29a00a92d16e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:10.508948 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8a571be2-5ad0-49c1-8975-29a00a92d16e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:10.510549 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e1c24e-2706-4bba-8e50-5bc216031f53 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:10.514467 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240a993a-dc77-4984-bf2f-8a01b885d50e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:10.515075 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240a993a-dc77-4984-bf2f-8a01b885d50e service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:10.515352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240a993a-dc77-4984-bf2f-8a01b885d50e service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:10.515631 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240a993a-dc77-4984-bf2f-8a01b885d50e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:10.518104 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-240a993a-dc77-4984-bf2f-8a01b885d50e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:10.576036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a571be2-5ad0-49c1-8975-29a00a92d16e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:10.577586 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240a993a-dc77-4984-bf2f-8a01b885d50e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:10.577930 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a571be2-5ad0-49c1-8975-29a00a92d16e service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:10.579103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240a993a-dc77-4984-bf2f-8a01b885d50e service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:10.588565 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1018/2061] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:10 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:10.588730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1044/2062] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:10 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:10.589290 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240a993a-dc77-4984-bf2f-8a01b885d50e service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:10.590624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-240a993a-dc77-4984-bf2f-8a01b885d50e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:10.595295 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a571be2-5ad0-49c1-8975-29a00a92d16e service cinder] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:10.600451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8a571be2-5ad0-49c1-8975-29a00a92d16e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:10.662596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240a993a-dc77-4984-bf2f-8a01b885d50e service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:10.667459 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43d6ea4-4f10-4aba-ac4b-c6618da2ab60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:10.668176 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43d6ea4-4f10-4aba-ac4b-c6618da2ab60 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:10.668579 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43d6ea4-4f10-4aba-ac4b-c6618da2ab60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:10.668857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43d6ea4-4f10-4aba-ac4b-c6618da2ab60 admin admin] PATH_INFO: `/v3/users/029d1350d2ec488c8a69eaa749b9548c` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:10.670017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43d6ea4-4f10-4aba-ac4b-c6618da2ab60 admin admin] RBAC: Authorizing `identity:delete_user(user_id=029d1350d2ec488c8a69eaa749b9548c)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:10.672007 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43d6ea4-4f10-4aba-ac4b-c6618da2ab60 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:10.676484 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a571be2-5ad0-49c1-8975-29a00a92d16e service cinder] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:10.686873 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d62781-0991-44a8-bad5-22753d0201f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:10.687796 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d62781-0991-44a8-bad5-22753d0201f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:10.688073 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d62781-0991-44a8-bad5-22753d0201f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:10.688339 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d62781-0991-44a8-bad5-22753d0201f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:10.689731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d62781-0991-44a8-bad5-22753d0201f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:10.698745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d62781-0991-44a8-bad5-22753d0201f8 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:10.709823 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1019/2063] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:10 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:10.711114 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d62781-0991-44a8-bad5-22753d0201f8 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:10.712598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-b1d62781-0991-44a8-bad5-22753d0201f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:10.724878 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d62781-0991-44a8-bad5-22753d0201f8 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:10.729752 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2742cce6-e548-4217-b50b-d1143e6bb18d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:10.730361 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2742cce6-e548-4217-b50b-d1143e6bb18d service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:10.730659 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2742cce6-e548-4217-b50b-d1143e6bb18d service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:10.730930 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2742cce6-e548-4217-b50b-d1143e6bb18d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:10.734745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2742cce6-e548-4217-b50b-d1143e6bb18d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:10.748226 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:10.793421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2742cce6-e548-4217-b50b-d1143e6bb18d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:10.794877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2742cce6-e548-4217-b50b-d1143e6bb18d service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:10.805023 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1020/2064] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:10 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:10.805818 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2742cce6-e548-4217-b50b-d1143e6bb18d service cinder] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:10.816987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2742cce6-e548-4217-b50b-d1143e6bb18d service cinder] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:10.822256 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab7e6e1-521a-4209-bc7c-89c1ff181f82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:10.822917 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab7e6e1-521a-4209-bc7c-89c1ff181f82 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:10.823214 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab7e6e1-521a-4209-bc7c-89c1ff181f82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:10.823431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab7e6e1-521a-4209-bc7c-89c1ff181f82 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:10.824068 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab7e6e1-521a-4209-bc7c-89c1ff181f82 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:10.825814 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ab7e6e1-521a-4209-bc7c-89c1ff181f82 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:10.835884 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1021/2065] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:10 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:10.840032 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab7e6e1-521a-4209-bc7c-89c1ff181f82 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:10.849703 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab7e6e1-521a-4209-bc7c-89c1ff181f82 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:10.854710 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3777862-4554-4f85-ad49-eb103b9841b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:10.855333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3777862-4554-4f85-ad49-eb103b9841b3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:10.855620 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3777862-4554-4f85-ad49-eb103b9841b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:10.855895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3777862-4554-4f85-ad49-eb103b9841b3 admin admin] PATH_INFO: `/v3/projects/e638ad4f53774de0a0d9925584015227/users/c8d53d49270c4fd9ada064582c793b71/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:10.873053 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3777862-4554-4f85-ad49-eb103b9841b3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e638ad4f53774de0a0d9925584015227, user_id=c8d53d49270c4fd9ada064582c793b71, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:10.878516 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3777862-4554-4f85-ad49-eb103b9841b3 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:10.911361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e43d6ea4-4f10-4aba-ac4b-c6618da2ab60 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '029d1350d2ec488c8a69eaa749b9548c'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:10.982155 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1022/2066] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:10 2022] PUT /identity/v3/projects/e638ad4f53774de0a0d9925584015227/users/c8d53d49270c4fd9ada064582c793b71/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:10.994719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53698e4f-ecb6-4f39-85bc-0e69ebe3576c None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:10.995093 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53698e4f-ecb6-4f39-85bc-0e69ebe3576c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:10.995372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53698e4f-ecb6-4f39-85bc-0e69ebe3576c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:10.998868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e43d6ea4-4f10-4aba-ac4b-c6618da2ab60 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '029d1350d2ec488c8a69eaa749b9548c'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:11.013051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e43d6ea4-4f10-4aba-ac4b-c6618da2ab60 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '029d1350d2ec488c8a69eaa749b9548c'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:11.014563 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e43d6ea4-4f10-4aba-ac4b-c6618da2ab60 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '029d1350d2ec488c8a69eaa749b9548c'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:11.022359 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e43d6ea4-4f10-4aba-ac4b-c6618da2ab60 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '029d1350d2ec488c8a69eaa749b9548c'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:11.024246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-53698e4f-ecb6-4f39-85bc-0e69ebe3576c None None] MFA Rules not processed for user `c8d53d49270c4fd9ada064582c793b71`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:11.025554 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-53698e4f-ecb6-4f39-85bc-0e69ebe3576c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:11.028971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1045/2067] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:10 2022] DELETE /identity/v3/users/029d1350d2ec488c8a69eaa749b9548c => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:11.047478 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:11.121512 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1023/2068] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:10 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:11.125877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43d6ea4-4f10-4aba-ac4b-c6618da2ab60 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:11.128581 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-e43d6ea4-4f10-4aba-ac4b-c6618da2ab60 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:11.213946 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43d6ea4-4f10-4aba-ac4b-c6618da2ab60 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:11.219271 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ee30c1-8d41-432b-bf25-4466baaebb57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:11.219882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ee30c1-8d41-432b-bf25-4466baaebb57 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:11.220820 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ee30c1-8d41-432b-bf25-4466baaebb57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:11.220820 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ee30c1-8d41-432b-bf25-4466baaebb57 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:11.221246 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ee30c1-8d41-432b-bf25-4466baaebb57 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:11.223206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ee30c1-8d41-432b-bf25-4466baaebb57 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:11.350232 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1046/2069] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:11 2022] POST /identity/v3/projects => generated 385 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:11.354446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53698e4f-ecb6-4f39-85bc-0e69ebe3576c None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:11.457949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53698e4f-ecb6-4f39-85bc-0e69ebe3576c None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:11.512803 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2a720d-ca4f-4af1-8c23-d1daa3e09e66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:11.513332 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2a720d-ca4f-4af1-8c23-d1daa3e09e66 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:11.513611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2a720d-ca4f-4af1-8c23-d1daa3e09e66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:11.513883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2a720d-ca4f-4af1-8c23-d1daa3e09e66 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:11.514560 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2a720d-ca4f-4af1-8c23-d1daa3e09e66 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:11.516472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2a720d-ca4f-4af1-8c23-d1daa3e09e66 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:11.562157 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ee30c1-8d41-432b-bf25-4466baaebb57 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:11.564772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-94ee30c1-8d41-432b-bf25-4466baaebb57 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:11.599997 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1024/2070] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:11 2022] POST /identity/v3/users => generated 299 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:11.603332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ee30c1-8d41-432b-bf25-4466baaebb57 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:11.609170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2a720d-ca4f-4af1-8c23-d1daa3e09e66 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:11.612906 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8047173e-80a7-4f97-b23e-306ea2bfa607 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:11.614860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8047173e-80a7-4f97-b23e-306ea2bfa607 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:11.614860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8047173e-80a7-4f97-b23e-306ea2bfa607 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:11.615304 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8047173e-80a7-4f97-b23e-306ea2bfa607 admin admin] PATH_INFO: `/v3/projects/403ed8c254684a8e87ee70b92a2dbf11` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:11.620107 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8047173e-80a7-4f97-b23e-306ea2bfa607 admin admin] RBAC: Authorizing `identity:delete_project(project_id=403ed8c254684a8e87ee70b92a2dbf11)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:11.625040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8047173e-80a7-4f97-b23e-306ea2bfa607 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:11.635954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2a720d-ca4f-4af1-8c23-d1daa3e09e66 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:11.642972 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5fa65e-c0b9-440e-9c2a-0769492bc34d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:11.644332 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5fa65e-c0b9-440e-9c2a-0769492bc34d admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:11.644813 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5fa65e-c0b9-440e-9c2a-0769492bc34d admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:11.645315 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5fa65e-c0b9-440e-9c2a-0769492bc34d admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:11.646219 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5fa65e-c0b9-440e-9c2a-0769492bc34d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:11.648827 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5fa65e-c0b9-440e-9c2a-0769492bc34d admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:11.670784 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1025/2071] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:11 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:11.685138 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5fa65e-c0b9-440e-9c2a-0769492bc34d admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:11.685507 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:11.697447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5fa65e-c0b9-440e-9c2a-0769492bc34d admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:11.702589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94386c6e-6838-4028-92cf-8c3b79b445a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:11.703264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94386c6e-6838-4028-92cf-8c3b79b445a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:11.703551 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94386c6e-6838-4028-92cf-8c3b79b445a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:11.703825 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94386c6e-6838-4028-92cf-8c3b79b445a9 admin admin] PATH_INFO: `/v3/projects/9fba941c9bd64367859c9c1e80fea68e/users/a67a5d885da840299335c06f1c369f54/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:11.714448 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:11.741507 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94386c6e-6838-4028-92cf-8c3b79b445a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9fba941c9bd64367859c9c1e80fea68e, user_id=a67a5d885da840299335c06f1c369f54, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:11.744938 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94386c6e-6838-4028-92cf-8c3b79b445a9 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:11.857138 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1026/2072] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:11 2022] PUT /identity/v3/projects/9fba941c9bd64367859c9c1e80fea68e/users/a67a5d885da840299335c06f1c369f54/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:11.857797 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94386c6e-6838-4028-92cf-8c3b79b445a9 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:11.861430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-94386c6e-6838-4028-92cf-8c3b79b445a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:11.868896 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8047173e-80a7-4f97-b23e-306ea2bfa607 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '403ed8c254684a8e87ee70b92a2dbf11', 'request_id': 'req-8047173e-80a7-4f97-b23e-306ea2bfa607'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:11.921278 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94386c6e-6838-4028-92cf-8c3b79b445a9 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:11.941917 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8047173e-80a7-4f97-b23e-306ea2bfa607 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '403ed8c254684a8e87ee70b92a2dbf11', 'request_id': 'req-8047173e-80a7-4f97-b23e-306ea2bfa607'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:11.948408 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8047173e-80a7-4f97-b23e-306ea2bfa607 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '403ed8c254684a8e87ee70b92a2dbf11', 'request_id': 'req-8047173e-80a7-4f97-b23e-306ea2bfa607'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:11.960595 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8047173e-80a7-4f97-b23e-306ea2bfa607 admin admin] The token cache is being invalidate because project 403ed8c254684a8e87ee70b92a2dbf11 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:11.960883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8047173e-80a7-4f97-b23e-306ea2bfa607 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:11.961975 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8047173e-80a7-4f97-b23e-306ea2bfa607 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:11.963529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1047/2073] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:11 2022] DELETE /identity/v3/projects/403ed8c254684a8e87ee70b92a2dbf11 => generated 0 bytes in 402 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:11.965032 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8047173e-80a7-4f97-b23e-306ea2bfa607 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:11.966477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8047173e-80a7-4f97-b23e-306ea2bfa607 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:11.973818 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d23aca3-3807-45a0-87f3-541348ebd881 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:11.974460 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d23aca3-3807-45a0-87f3-541348ebd881 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:11.974740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d23aca3-3807-45a0-87f3-541348ebd881 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:11.975007 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d23aca3-3807-45a0-87f3-541348ebd881 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:11.977288 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-7d23aca3-3807-45a0-87f3-541348ebd881 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:12.035497 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8047173e-80a7-4f97-b23e-306ea2bfa607 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:12.039187 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d23aca3-3807-45a0-87f3-541348ebd881 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:12.040414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbdf8bd-4ddf-444b-9a6f-fead6ee4be6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:12.040640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d23aca3-3807-45a0-87f3-541348ebd881 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:12.041151 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbdf8bd-4ddf-444b-9a6f-fead6ee4be6f admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:12.041427 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbdf8bd-4ddf-444b-9a6f-fead6ee4be6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:12.041724 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbdf8bd-4ddf-444b-9a6f-fead6ee4be6f admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:12.042390 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbdf8bd-4ddf-444b-9a6f-fead6ee4be6f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:12.044031 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbdf8bd-4ddf-444b-9a6f-fead6ee4be6f admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:12.051280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1027/2074] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:11 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:12.052516 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3259b149-9e24-4948-8367-5f181f1bb5dc None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:12.052776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3259b149-9e24-4948-8367-5f181f1bb5dc None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:12.053044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3259b149-9e24-4948-8367-5f181f1bb5dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:12.055912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1048/2075] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:11 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:12.059964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbdf8bd-4ddf-444b-9a6f-fead6ee4be6f admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:12.070950 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbdf8bd-4ddf-444b-9a6f-fead6ee4be6f admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:12.075550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-3259b149-9e24-4948-8367-5f181f1bb5dc None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:12.075993 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea57402-c1d6-4895-902d-280e679c6b23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:12.076645 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea57402-c1d6-4895-902d-280e679c6b23 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:12.076945 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3259b149-9e24-4948-8367-5f181f1bb5dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:12.077060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea57402-c1d6-4895-902d-280e679c6b23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:12.077484 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea57402-c1d6-4895-902d-280e679c6b23 admin admin] PATH_INFO: `/v3/projects/9fba941c9bd64367859c9c1e80fea68e/users/a67a5d885da840299335c06f1c369f54/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:12.079461 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea57402-c1d6-4895-902d-280e679c6b23 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9fba941c9bd64367859c9c1e80fea68e, user_id=a67a5d885da840299335c06f1c369f54, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:12.082536 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea57402-c1d6-4895-902d-280e679c6b23 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:12.093165 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1028/2076] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:12.094783 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecde09b5-0308-4632-aa66-c866e14d3db0 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:12.095670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecde09b5-0308-4632-aa66-c866e14d3db0 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:12.096110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecde09b5-0308-4632-aa66-c866e14d3db0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:12.097670 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1049/2077] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:12 2022] PUT /identity/v3/projects/9fba941c9bd64367859c9c1e80fea68e/users/a67a5d885da840299335c06f1c369f54/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:12.107945 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bff3b5-2e4f-4e8d-9692-82a42c86dae6 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:12.108236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bff3b5-2e4f-4e8d-9692-82a42c86dae6 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:12.112061 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bff3b5-2e4f-4e8d-9692-82a42c86dae6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:12.124800 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-ecde09b5-0308-4632-aa66-c866e14d3db0 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:12.126431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ecde09b5-0308-4632-aa66-c866e14d3db0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:12.140593 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-47bff3b5-2e4f-4e8d-9692-82a42c86dae6 None None] MFA Rules not processed for user `a67a5d885da840299335c06f1c369f54`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:12.141874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-47bff3b5-2e4f-4e8d-9692-82a42c86dae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:12.198234 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1029/2078] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:12.199961 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecde09b5-0308-4632-aa66-c866e14d3db0 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:12.210400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecde09b5-0308-4632-aa66-c866e14d3db0 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:12.218088 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfd334f-0e4d-42eb-a3d8-48c269c422a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:12.218719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfd334f-0e4d-42eb-a3d8-48c269c422a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:12.219004 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfd334f-0e4d-42eb-a3d8-48c269c422a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:12.219278 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfd334f-0e4d-42eb-a3d8-48c269c422a5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:12.220441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfd334f-0e4d-42eb-a3d8-48c269c422a5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:03:12.220743 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfd334f-0e4d-42eb-a3d8-48c269c422a5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:12.222534 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfd334f-0e4d-42eb-a3d8-48c269c422a5 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:12.223502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1050/2079] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:12 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:12.233867 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bff3b5-2e4f-4e8d-9692-82a42c86dae6 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:12.234822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1030/2080] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:03:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:03:12.235252 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-47bff3b5-2e4f-4e8d-9692-82a42c86dae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:12.242009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfd334f-0e4d-42eb-a3d8-48c269c422a5 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:12.252415 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfd334f-0e4d-42eb-a3d8-48c269c422a5 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:12.257990 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4683ba75-690c-4f4b-b28a-8bea96934d05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:12.258623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4683ba75-690c-4f4b-b28a-8bea96934d05 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:12.258904 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4683ba75-690c-4f4b-b28a-8bea96934d05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:12.259236 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4683ba75-690c-4f4b-b28a-8bea96934d05 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:12.259846 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4683ba75-690c-4f4b-b28a-8bea96934d05 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:12.261758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4683ba75-690c-4f4b-b28a-8bea96934d05 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:12.295633 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bff3b5-2e4f-4e8d-9692-82a42c86dae6 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:12.299376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41523740-1744-41ab-ae69-a1a8d8075143 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:12.299972 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41523740-1744-41ab-ae69-a1a8d8075143 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:12.300500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41523740-1744-41ab-ae69-a1a8d8075143 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:12.300770 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41523740-1744-41ab-ae69-a1a8d8075143 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:12.309694 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41523740-1744-41ab-ae69-a1a8d8075143 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:12.311128 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41523740-1744-41ab-ae69-a1a8d8075143 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:12.320579 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1051/2081] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:12 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:03:12.326161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1031/2082] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:12 2022] POST /identity/v3/projects => generated 399 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:03:12.330977 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41523740-1744-41ab-ae69-a1a8d8075143 service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:12.337985 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:12.342490 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41523740-1744-41ab-ae69-a1a8d8075143 service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:12.400015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891e264e-e470-4e5a-ab5e-fd0eb6d1f3f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:12.400874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891e264e-e470-4e5a-ab5e-fd0eb6d1f3f7 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:12.400874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891e264e-e470-4e5a-ab5e-fd0eb6d1f3f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:12.401195 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891e264e-e470-4e5a-ab5e-fd0eb6d1f3f7 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:12.401773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891e264e-e470-4e5a-ab5e-fd0eb6d1f3f7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:12.403751 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891e264e-e470-4e5a-ab5e-fd0eb6d1f3f7 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:12.413279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:12.506096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1052/2083] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:12 2022] POST /identity/v3/users => generated 300 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:03:12.507449 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4683ba75-690c-4f4b-b28a-8bea96934d05 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:12.518136 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4683ba75-690c-4f4b-b28a-8bea96934d05 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:12.523155 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e57a47-992e-403d-bb28-dd854772e573 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:12.523867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e57a47-992e-403d-bb28-dd854772e573 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:12.524094 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e57a47-992e-403d-bb28-dd854772e573 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:12.524368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e57a47-992e-403d-bb28-dd854772e573 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:12.525012 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e57a47-992e-403d-bb28-dd854772e573 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:12.526637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e57a47-992e-403d-bb28-dd854772e573 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:12.537538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1032/2084] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:12.540962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891e264e-e470-4e5a-ab5e-fd0eb6d1f3f7 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:12.558242 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891e264e-e470-4e5a-ab5e-fd0eb6d1f3f7 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:12.558242 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff0544f-4c1f-4f1a-87a3-ec204a4d9085 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:12.558242 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff0544f-4c1f-4f1a-87a3-ec204a4d9085 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:12.558242 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff0544f-4c1f-4f1a-87a3-ec204a4d9085 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:12.558674 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff0544f-4c1f-4f1a-87a3-ec204a4d9085 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:12.558674 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff0544f-4c1f-4f1a-87a3-ec204a4d9085 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:12.559969 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff0544f-4c1f-4f1a-87a3-ec204a4d9085 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:12.570041 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1053/2085] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:12.574107 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e57a47-992e-403d-bb28-dd854772e573 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:12.587083 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e57a47-992e-403d-bb28-dd854772e573 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:12.591394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f6aa76-4faf-415a-9f3b-1d58de7dba2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:12.592059 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f6aa76-4faf-415a-9f3b-1d58de7dba2b admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:12.592443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f6aa76-4faf-415a-9f3b-1d58de7dba2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:12.592599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f6aa76-4faf-415a-9f3b-1d58de7dba2b admin admin] PATH_INFO: `/v3/projects/b5f5f78825d5490bbe170255f4c85070/users/063aa7c6b04c4ee99f9f5f20a872de9a/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:12.631508 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f6aa76-4faf-415a-9f3b-1d58de7dba2b admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5f5f78825d5490bbe170255f4c85070, user_id=063aa7c6b04c4ee99f9f5f20a872de9a, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:12.634364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f6aa76-4faf-415a-9f3b-1d58de7dba2b admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:12.698527 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1033/2086] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:12 2022] PUT /identity/v3/projects/b5f5f78825d5490bbe170255f4c85070/users/063aa7c6b04c4ee99f9f5f20a872de9a/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:12.709611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bac9d9-a4d7-4fa8-981c-87b409b0a1e0 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:12.709827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bac9d9-a4d7-4fa8-981c-87b409b0a1e0 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:12.710104 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bac9d9-a4d7-4fa8-981c-87b409b0a1e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:12.756984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-a9bac9d9-a4d7-4fa8-981c-87b409b0a1e0 None None] MFA Rules not processed for user `063aa7c6b04c4ee99f9f5f20a872de9a`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:12.763231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a9bac9d9-a4d7-4fa8-981c-87b409b0a1e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:12.860829 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1054/2087] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:12 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:12.864605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f6aa76-4faf-415a-9f3b-1d58de7dba2b admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:12.953182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f6aa76-4faf-415a-9f3b-1d58de7dba2b admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:12.959779 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73c8409-f067-4d6d-8475-123640c9da89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:12.960994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73c8409-f067-4d6d-8475-123640c9da89 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:12.960994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73c8409-f067-4d6d-8475-123640c9da89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:12.960994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73c8409-f067-4d6d-8475-123640c9da89 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:12.961743 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73c8409-f067-4d6d-8475-123640c9da89 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:12.964231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73c8409-f067-4d6d-8475-123640c9da89 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:12.993774 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:13.057981 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1034/2088] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:12 2022] POST /identity/v3/projects => generated 401 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:13.062067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9bac9d9-a4d7-4fa8-981c-87b409b0a1e0 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:13.073482 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9bac9d9-a4d7-4fa8-981c-87b409b0a1e0 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:13.131757 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a811b8f-68f3-4bee-a775-3d9fe437b31d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:13.132360 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a811b8f-68f3-4bee-a775-3d9fe437b31d admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:13.132637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a811b8f-68f3-4bee-a775-3d9fe437b31d admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:13.132907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a811b8f-68f3-4bee-a775-3d9fe437b31d admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:13.133580 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a811b8f-68f3-4bee-a775-3d9fe437b31d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:13.135511 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a811b8f-68f3-4bee-a775-3d9fe437b31d admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:13.255703 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1055/2089] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:13 2022] POST /identity/v3/users => generated 307 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:13.258787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73c8409-f067-4d6d-8475-123640c9da89 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:13.269194 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73c8409-f067-4d6d-8475-123640c9da89 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:13.273882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174f3dae-6d94-4908-9a78-7179d5e46d09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:13.274522 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174f3dae-6d94-4908-9a78-7179d5e46d09 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:13.274802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174f3dae-6d94-4908-9a78-7179d5e46d09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:13.275069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174f3dae-6d94-4908-9a78-7179d5e46d09 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:13.275983 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174f3dae-6d94-4908-9a78-7179d5e46d09 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:13.277434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-174f3dae-6d94-4908-9a78-7179d5e46d09 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:13.291867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1035/2090] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:13 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:03:13.293080 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a811b8f-68f3-4bee-a775-3d9fe437b31d admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:13.303964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a811b8f-68f3-4bee-a775-3d9fe437b31d admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:13.308586 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696fddb1-63d2-4c07-a392-d86dcd31d5cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:13.309226 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696fddb1-63d2-4c07-a392-d86dcd31d5cf admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:13.309505 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696fddb1-63d2-4c07-a392-d86dcd31d5cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:13.309771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-696fddb1-63d2-4c07-a392-d86dcd31d5cf admin admin] PATH_INFO: `/v3/projects/a4c2dcf8f82b4dfaa46dca9d6851bac1/users/e4673fba9c9049b5b6106e70decb1082/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:13.325731 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696fddb1-63d2-4c07-a392-d86dcd31d5cf admin admin] RBAC: Authorizing `identity:create_grant(project_id=a4c2dcf8f82b4dfaa46dca9d6851bac1, user_id=e4673fba9c9049b5b6106e70decb1082, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:13.328444 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-696fddb1-63d2-4c07-a392-d86dcd31d5cf admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:13.382169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:13.405662 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1056/2091] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:13 2022] PUT /identity/v3/projects/a4c2dcf8f82b4dfaa46dca9d6851bac1/users/e4673fba9c9049b5b6106e70decb1082/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:13.411311 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174f3dae-6d94-4908-9a78-7179d5e46d09 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:13.422221 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174f3dae-6d94-4908-9a78-7179d5e46d09 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:13.480888 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b974aeb8-0e34-4f9e-91fb-8c47c8bc7c51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:13.481525 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b974aeb8-0e34-4f9e-91fb-8c47c8bc7c51 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:13.481805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b974aeb8-0e34-4f9e-91fb-8c47c8bc7c51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:13.482076 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b974aeb8-0e34-4f9e-91fb-8c47c8bc7c51 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:13.482781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b974aeb8-0e34-4f9e-91fb-8c47c8bc7c51 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:13.484402 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b974aeb8-0e34-4f9e-91fb-8c47c8bc7c51 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:13.494897 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1036/2092] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:13 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:13.499121 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696fddb1-63d2-4c07-a392-d86dcd31d5cf admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:13.509558 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-696fddb1-63d2-4c07-a392-d86dcd31d5cf admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:13.518348 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a99460-07b3-4ddf-9b87-57d6bb706628 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:13.518348 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a99460-07b3-4ddf-9b87-57d6bb706628 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:13.518348 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a99460-07b3-4ddf-9b87-57d6bb706628 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:13.518348 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a99460-07b3-4ddf-9b87-57d6bb706628 admin admin] PATH_INFO: `/v3/projects/a4c2dcf8f82b4dfaa46dca9d6851bac1/users/e4673fba9c9049b5b6106e70decb1082/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:13.518680 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a99460-07b3-4ddf-9b87-57d6bb706628 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a4c2dcf8f82b4dfaa46dca9d6851bac1, user_id=e4673fba9c9049b5b6106e70decb1082, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:13.519924 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a99460-07b3-4ddf-9b87-57d6bb706628 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:13.529916 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1057/2093] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:13 2022] PUT /identity/v3/projects/a4c2dcf8f82b4dfaa46dca9d6851bac1/users/e4673fba9c9049b5b6106e70decb1082/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:13.541782 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897f412d-6f4a-4c0b-9e18-b190460f469d None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:13.542066 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897f412d-6f4a-4c0b-9e18-b190460f469d None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:13.542359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897f412d-6f4a-4c0b-9e18-b190460f469d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:13.570607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-897f412d-6f4a-4c0b-9e18-b190460f469d None None] MFA Rules not processed for user `e4673fba9c9049b5b6106e70decb1082`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:13.571819 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-897f412d-6f4a-4c0b-9e18-b190460f469d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:13.647822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1037/2094] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:13 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:13.659899 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a99460-07b3-4ddf-9b87-57d6bb706628 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:13.714160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a99460-07b3-4ddf-9b87-57d6bb706628 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:13.717835 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77f67da-7e95-4c39-8b26-52e88e3a4432 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:13.718466 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77f67da-7e95-4c39-8b26-52e88e3a4432 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:13.718747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77f67da-7e95-4c39-8b26-52e88e3a4432 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:13.719019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77f67da-7e95-4c39-8b26-52e88e3a4432 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:13.728527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77f67da-7e95-4c39-8b26-52e88e3a4432 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:13.729943 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c77f67da-7e95-4c39-8b26-52e88e3a4432 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:13.739865 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1058/2095] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:13 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:13.767499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:13.854402 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897f412d-6f4a-4c0b-9e18-b190460f469d None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:13.920721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897f412d-6f4a-4c0b-9e18-b190460f469d None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:13.925936 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18fb42e-6dec-4e65-8bfb-9d96b3ab7ae5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:13.926593 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18fb42e-6dec-4e65-8bfb-9d96b3ab7ae5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:13.926874 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18fb42e-6dec-4e65-8bfb-9d96b3ab7ae5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:13.927144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18fb42e-6dec-4e65-8bfb-9d96b3ab7ae5 admin admin] PATH_INFO: `/v3/users/c8d53d49270c4fd9ada064582c793b71` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:13.928341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18fb42e-6dec-4e65-8bfb-9d96b3ab7ae5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c8d53d49270c4fd9ada064582c793b71)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:13.930265 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18fb42e-6dec-4e65-8bfb-9d96b3ab7ae5 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:13.962349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77f67da-7e95-4c39-8b26-52e88e3a4432 service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:13.972049 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77f67da-7e95-4c39-8b26-52e88e3a4432 service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:13.975702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a97df37-a6b9-4865-9463-f4dbc2b6fc3d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:13.976286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a97df37-a6b9-4865-9463-f4dbc2b6fc3d service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:13.976566 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a97df37-a6b9-4865-9463-f4dbc2b6fc3d service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:13.976840 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a97df37-a6b9-4865-9463-f4dbc2b6fc3d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:14.035583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a97df37-a6b9-4865-9463-f4dbc2b6fc3d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:14.037437 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a97df37-a6b9-4865-9463-f4dbc2b6fc3d service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:14.046747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1059/2096] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:13 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:14.048465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d18fb42e-6dec-4e65-8bfb-9d96b3ab7ae5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c8d53d49270c4fd9ada064582c793b71'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:14.051735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:14.113227 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d18fb42e-6dec-4e65-8bfb-9d96b3ab7ae5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c8d53d49270c4fd9ada064582c793b71'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:14.114083 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d18fb42e-6dec-4e65-8bfb-9d96b3ab7ae5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c8d53d49270c4fd9ada064582c793b71'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:14.129100 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d18fb42e-6dec-4e65-8bfb-9d96b3ab7ae5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c8d53d49270c4fd9ada064582c793b71'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:14.138534 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d18fb42e-6dec-4e65-8bfb-9d96b3ab7ae5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c8d53d49270c4fd9ada064582c793b71'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:14.147207 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1038/2097] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:13 2022] DELETE /identity/v3/users/c8d53d49270c4fd9ada064582c793b71 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:14.149409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:14.151542 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9bb0d6-78a5-4f0a-95f5-acd38aa9e61b None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:14.156925 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9bb0d6-78a5-4f0a-95f5-acd38aa9e61b None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:14.156925 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9bb0d6-78a5-4f0a-95f5-acd38aa9e61b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:14.176373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-8b9bb0d6-78a5-4f0a-95f5-acd38aa9e61b None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:14.184404 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8b9bb0d6-78a5-4f0a-95f5-acd38aa9e61b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:14.275646 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1060/2098] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:14.286305 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18fb42e-6dec-4e65-8bfb-9d96b3ab7ae5 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:14.287765 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d18fb42e-6dec-4e65-8bfb-9d96b3ab7ae5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:14.348468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18fb42e-6dec-4e65-8bfb-9d96b3ab7ae5 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:14.352119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ded8824-f596-48af-8934-3ca83223eab6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:14.352769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ded8824-f596-48af-8934-3ca83223eab6 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:14.353164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ded8824-f596-48af-8934-3ca83223eab6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:14.353492 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ded8824-f596-48af-8934-3ca83223eab6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:14.363674 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ded8824-f596-48af-8934-3ca83223eab6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:14.372256 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ded8824-f596-48af-8934-3ca83223eab6 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:14.374934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1039/2099] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:14.441849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:14.456350 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9bb0d6-78a5-4f0a-95f5-acd38aa9e61b None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:14.457715 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8b9bb0d6-78a5-4f0a-95f5-acd38aa9e61b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:14.478059 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9bb0d6-78a5-4f0a-95f5-acd38aa9e61b None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:14.483239 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4206fb93-8160-4f9b-8cc2-cd0792aa140c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:14.484130 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4206fb93-8160-4f9b-8cc2-cd0792aa140c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:14.484130 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4206fb93-8160-4f9b-8cc2-cd0792aa140c admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:14.484402 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4206fb93-8160-4f9b-8cc2-cd0792aa140c admin admin] PATH_INFO: `/v3/users/ecdf2746343945749f54839322419dbe` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:14.486249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4206fb93-8160-4f9b-8cc2-cd0792aa140c admin admin] RBAC: Authorizing `identity:delete_user(user_id=ecdf2746343945749f54839322419dbe)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:14.488356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4206fb93-8160-4f9b-8cc2-cd0792aa140c admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:14.691394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:14.740393 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-4206fb93-8160-4f9b-8cc2-cd0792aa140c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ecdf2746343945749f54839322419dbe'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:14.809854 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-4206fb93-8160-4f9b-8cc2-cd0792aa140c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ecdf2746343945749f54839322419dbe'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:14.822504 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-4206fb93-8160-4f9b-8cc2-cd0792aa140c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ecdf2746343945749f54839322419dbe'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:14.823138 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-4206fb93-8160-4f9b-8cc2-cd0792aa140c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ecdf2746343945749f54839322419dbe'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:14.830720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-4206fb93-8160-4f9b-8cc2-cd0792aa140c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ecdf2746343945749f54839322419dbe'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:14.837007 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1061/2100] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:14 2022] DELETE /identity/v3/users/ecdf2746343945749f54839322419dbe => generated 0 bytes in 382 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:14.842215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5586478-d3ed-424a-b310-a92514a4a590 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:14.842698 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5586478-d3ed-424a-b310-a92514a4a590 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:14.843081 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5586478-d3ed-424a-b310-a92514a4a590 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:14.873800 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-c5586478-d3ed-424a-b310-a92514a4a590 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:14.875286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-c5586478-d3ed-424a-b310-a92514a4a590 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:14.907205 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4206fb93-8160-4f9b-8cc2-cd0792aa140c admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:14.909766 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-4206fb93-8160-4f9b-8cc2-cd0792aa140c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:14.968662 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1040/2101] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:14.979365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5586478-d3ed-424a-b310-a92514a4a590 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:14.980842 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-c5586478-d3ed-424a-b310-a92514a4a590 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:14.999573 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4206fb93-8160-4f9b-8cc2-cd0792aa140c admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:15.006057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7574220d-1c29-4817-86d0-0bd0ff5ae435 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:15.006728 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7574220d-1c29-4817-86d0-0bd0ff5ae435 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:15.007005 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7574220d-1c29-4817-86d0-0bd0ff5ae435 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:15.007279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7574220d-1c29-4817-86d0-0bd0ff5ae435 admin admin] PATH_INFO: `/v3/projects/e638ad4f53774de0a0d9925584015227` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:15.008415 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7574220d-1c29-4817-86d0-0bd0ff5ae435 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e638ad4f53774de0a0d9925584015227)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:15.010434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7574220d-1c29-4817-86d0-0bd0ff5ae435 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:15.046229 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5586478-d3ed-424a-b310-a92514a4a590 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:15.050991 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba134c6-6aa0-48f7-b92a-2f2411fe2946 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:15.051603 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba134c6-6aa0-48f7-b92a-2f2411fe2946 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:15.052188 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba134c6-6aa0-48f7-b92a-2f2411fe2946 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:15.052188 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba134c6-6aa0-48f7-b92a-2f2411fe2946 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:15.061862 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba134c6-6aa0-48f7-b92a-2f2411fe2946 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:15.063458 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba134c6-6aa0-48f7-b92a-2f2411fe2946 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:15.073095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1041/2102] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:15.174572 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7574220d-1c29-4817-86d0-0bd0ff5ae435 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e638ad4f53774de0a0d9925584015227', 'request_id': 'req-7574220d-1c29-4817-86d0-0bd0ff5ae435'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:15.285538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7574220d-1c29-4817-86d0-0bd0ff5ae435 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e638ad4f53774de0a0d9925584015227', 'request_id': 'req-7574220d-1c29-4817-86d0-0bd0ff5ae435'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:15.292812 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7574220d-1c29-4817-86d0-0bd0ff5ae435 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e638ad4f53774de0a0d9925584015227', 'request_id': 'req-7574220d-1c29-4817-86d0-0bd0ff5ae435'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:15.309259 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7574220d-1c29-4817-86d0-0bd0ff5ae435 admin admin] The token cache is being invalidate because project e638ad4f53774de0a0d9925584015227 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:15.309553 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7574220d-1c29-4817-86d0-0bd0ff5ae435 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:15.310219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7574220d-1c29-4817-86d0-0bd0ff5ae435 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:15.311814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1062/2103] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:14 2022] DELETE /identity/v3/projects/e638ad4f53774de0a0d9925584015227 => generated 0 bytes in 405 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:15.316175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba134c6-6aa0-48f7-b92a-2f2411fe2946 service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:15.325933 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba134c6-6aa0-48f7-b92a-2f2411fe2946 service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:15.383709 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:15.385261 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45793c3-071c-4f75-9830-bf6bda17804c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:15.385982 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45793c3-071c-4f75-9830-bf6bda17804c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:15.386260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45793c3-071c-4f75-9830-bf6bda17804c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:15.386567 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45793c3-071c-4f75-9830-bf6bda17804c admin admin] PATH_INFO: `/v3/users/a67a5d885da840299335c06f1c369f54` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:15.387682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45793c3-071c-4f75-9830-bf6bda17804c admin admin] RBAC: Authorizing `identity:delete_user(user_id=a67a5d885da840299335c06f1c369f54)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:15.389651 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e45793c3-071c-4f75-9830-bf6bda17804c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:15.416319 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:15.543540 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e45793c3-071c-4f75-9830-bf6bda17804c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a67a5d885da840299335c06f1c369f54'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:15.593036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e45793c3-071c-4f75-9830-bf6bda17804c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a67a5d885da840299335c06f1c369f54'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:15.593862 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e45793c3-071c-4f75-9830-bf6bda17804c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a67a5d885da840299335c06f1c369f54'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:15.607440 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e45793c3-071c-4f75-9830-bf6bda17804c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a67a5d885da840299335c06f1c369f54'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:15.615309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e45793c3-071c-4f75-9830-bf6bda17804c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a67a5d885da840299335c06f1c369f54'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:15.621337 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1042/2104] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:15 2022] DELETE /identity/v3/users/a67a5d885da840299335c06f1c369f54 => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:15.648781 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7574220d-1c29-4817-86d0-0bd0ff5ae435 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:15.650303 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-7574220d-1c29-4817-86d0-0bd0ff5ae435 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:15.663058 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:15.664587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7574220d-1c29-4817-86d0-0bd0ff5ae435 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:15.736809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd82835d-4557-4fe8-8b44-6f5a505dfe3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:15.737457 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd82835d-4557-4fe8-8b44-6f5a505dfe3f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:15.737745 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd82835d-4557-4fe8-8b44-6f5a505dfe3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:15.738012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd82835d-4557-4fe8-8b44-6f5a505dfe3f admin admin] PATH_INFO: `/v3/projects/3c0e6b9e809c43a8975c25124b157c6c` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:15.739223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd82835d-4557-4fe8-8b44-6f5a505dfe3f admin admin] RBAC: Authorizing `identity:delete_project(project_id=3c0e6b9e809c43a8975c25124b157c6c)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:15.741158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd82835d-4557-4fe8-8b44-6f5a505dfe3f admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:15.903974 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-dd82835d-4557-4fe8-8b44-6f5a505dfe3f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3c0e6b9e809c43a8975c25124b157c6c', 'request_id': 'req-dd82835d-4557-4fe8-8b44-6f5a505dfe3f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:15.947499 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-dd82835d-4557-4fe8-8b44-6f5a505dfe3f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3c0e6b9e809c43a8975c25124b157c6c', 'request_id': 'req-dd82835d-4557-4fe8-8b44-6f5a505dfe3f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:15.954701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-dd82835d-4557-4fe8-8b44-6f5a505dfe3f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3c0e6b9e809c43a8975c25124b157c6c', 'request_id': 'req-dd82835d-4557-4fe8-8b44-6f5a505dfe3f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:15.971938 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-dd82835d-4557-4fe8-8b44-6f5a505dfe3f admin admin] The token cache is being invalidate because project 3c0e6b9e809c43a8975c25124b157c6c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:15.972455 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-dd82835d-4557-4fe8-8b44-6f5a505dfe3f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:15.973253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-dd82835d-4557-4fe8-8b44-6f5a505dfe3f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:15.975552 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1063/2105] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:15 2022] DELETE /identity/v3/projects/3c0e6b9e809c43a8975c25124b157c6c => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:15.983683 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ee1914-7fa0-4819-b203-35914f40c616 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:15.983683 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ee1914-7fa0-4819-b203-35914f40c616 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:15.983683 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ee1914-7fa0-4819-b203-35914f40c616 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:15.999224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:16.005989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-e3ee1914-7fa0-4819-b203-35914f40c616 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:16.011844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e3ee1914-7fa0-4819-b203-35914f40c616 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:16.067903 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1043/2106] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:16.076970 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8edaab-1035-4685-ab29-aca7ab86eacb None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:16.077425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8edaab-1035-4685-ab29-aca7ab86eacb None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:16.077847 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8edaab-1035-4685-ab29-aca7ab86eacb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:16.080014 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:16.093825 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ee1914-7fa0-4819-b203-35914f40c616 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:16.096033 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e3ee1914-7fa0-4819-b203-35914f40c616 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:16.106421 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-8f8edaab-1035-4685-ab29-aca7ab86eacb None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:16.107760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8f8edaab-1035-4685-ab29-aca7ab86eacb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:16.110865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ee1914-7fa0-4819-b203-35914f40c616 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:16.115887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4ff72e-9229-4747-a1c1-3ef83bef699f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:16.116528 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4ff72e-9229-4747-a1c1-3ef83bef699f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:16.116808 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4ff72e-9229-4747-a1c1-3ef83bef699f admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:16.117451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4ff72e-9229-4747-a1c1-3ef83bef699f admin admin] PATH_INFO: `/v3/projects/9fba941c9bd64367859c9c1e80fea68e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:16.118235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1064/2107] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:16.121753 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4ff72e-9229-4747-a1c1-3ef83bef699f admin admin] RBAC: Authorizing `identity:delete_project(project_id=9fba941c9bd64367859c9c1e80fea68e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:16.129386 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4ff72e-9229-4747-a1c1-3ef83bef699f admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:16.129830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8edaab-1035-4685-ab29-aca7ab86eacb None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:16.138137 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8edaab-1035-4685-ab29-aca7ab86eacb None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:16.144551 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcc4aac-2eb6-415f-b5e1-944809631b9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:16.145301 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcc4aac-2eb6-415f-b5e1-944809631b9c admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:16.145579 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcc4aac-2eb6-415f-b5e1-944809631b9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:16.145848 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcc4aac-2eb6-415f-b5e1-944809631b9c admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:16.146542 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcc4aac-2eb6-415f-b5e1-944809631b9c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:03:16.147649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcc4aac-2eb6-415f-b5e1-944809631b9c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:16.148724 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffcc4aac-2eb6-415f-b5e1-944809631b9c admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:16.161812 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1065/2108] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:03:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 14:03:16.166084 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcc4aac-2eb6-415f-b5e1-944809631b9c admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:16.177497 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcc4aac-2eb6-415f-b5e1-944809631b9c admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:16.182120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1eb6a10-790f-46b4-8cab-1ce3382193d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:16.182670 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1eb6a10-790f-46b4-8cab-1ce3382193d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:16.182948 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1eb6a10-790f-46b4-8cab-1ce3382193d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:16.183219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1eb6a10-790f-46b4-8cab-1ce3382193d4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:16.183837 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1eb6a10-790f-46b4-8cab-1ce3382193d4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:16.185696 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1eb6a10-790f-46b4-8cab-1ce3382193d4 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:16.280405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1066/2109] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:16 2022] POST /identity/v3/projects => generated 377 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:03:16.280571 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-6d4ff72e-9229-4747-a1c1-3ef83bef699f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9fba941c9bd64367859c9c1e80fea68e', 'request_id': 'req-6d4ff72e-9229-4747-a1c1-3ef83bef699f'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:16.284914 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1eb6a10-790f-46b4-8cab-1ce3382193d4 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:16.351957 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1eb6a10-790f-46b4-8cab-1ce3382193d4 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:16.356603 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20ed399-1ed2-4acc-aac4-e5f6f0dab8b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:16.357288 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20ed399-1ed2-4acc-aac4-e5f6f0dab8b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:16.357567 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20ed399-1ed2-4acc-aac4-e5f6f0dab8b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:16.357838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20ed399-1ed2-4acc-aac4-e5f6f0dab8b2 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:16.358519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20ed399-1ed2-4acc-aac4-e5f6f0dab8b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:16.360973 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20ed399-1ed2-4acc-aac4-e5f6f0dab8b2 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:16.381318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-6d4ff72e-9229-4747-a1c1-3ef83bef699f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9fba941c9bd64367859c9c1e80fea68e', 'request_id': 'req-6d4ff72e-9229-4747-a1c1-3ef83bef699f'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:16.387306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-6d4ff72e-9229-4747-a1c1-3ef83bef699f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9fba941c9bd64367859c9c1e80fea68e', 'request_id': 'req-6d4ff72e-9229-4747-a1c1-3ef83bef699f'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:16.400299 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-6d4ff72e-9229-4747-a1c1-3ef83bef699f admin admin] The token cache is being invalidate because project 9fba941c9bd64367859c9c1e80fea68e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:16.400590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-6d4ff72e-9229-4747-a1c1-3ef83bef699f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:16.401275 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-6d4ff72e-9229-4747-a1c1-3ef83bef699f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:16.402849 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1044/2110] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:16 2022] DELETE /identity/v3/projects/9fba941c9bd64367859c9c1e80fea68e => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:16.408788 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fe48bb-281e-4099-9340-fb036052975e None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:16.409069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fe48bb-281e-4099-9340-fb036052975e None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:16.409338 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3fe48bb-281e-4099-9340-fb036052975e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:16.430132 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-b3fe48bb-281e-4099-9340-fb036052975e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:16.431379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-b3fe48bb-281e-4099-9340-fb036052975e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:16.441054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1045/2111] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:16.448633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daba1fe1-7f0a-4887-92f9-f71c0e7ca70c None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:16.448980 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daba1fe1-7f0a-4887-92f9-f71c0e7ca70c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:16.449257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daba1fe1-7f0a-4887-92f9-f71c0e7ca70c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:16.468218 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1067/2112] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:16 2022] POST /identity/v3/users => generated 289 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:16.469120 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-daba1fe1-7f0a-4887-92f9-f71c0e7ca70c None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:16.470988 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-daba1fe1-7f0a-4887-92f9-f71c0e7ca70c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:16.472893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20ed399-1ed2-4acc-aac4-e5f6f0dab8b2 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:16.474602 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d20ed399-1ed2-4acc-aac4-e5f6f0dab8b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:16.484679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1046/2113] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:16.489446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daba1fe1-7f0a-4887-92f9-f71c0e7ca70c None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:16.496185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20ed399-1ed2-4acc-aac4-e5f6f0dab8b2 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:16.497346 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674515a4-50f5-45c0-a964-420bddeabbfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:16.498254 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674515a4-50f5-45c0-a964-420bddeabbfb admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:16.498569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674515a4-50f5-45c0-a964-420bddeabbfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:16.499026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674515a4-50f5-45c0-a964-420bddeabbfb admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:16.499684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674515a4-50f5-45c0-a964-420bddeabbfb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:16.500759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daba1fe1-7f0a-4887-92f9-f71c0e7ca70c None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:16.502221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674515a4-50f5-45c0-a964-420bddeabbfb admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:16.508137 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:16.510278 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1edee3c-4461-4a0c-bf29-0c41dcdde781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:16.510937 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1edee3c-4461-4a0c-bf29-0c41dcdde781 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:16.515053 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1edee3c-4461-4a0c-bf29-0c41dcdde781 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:16.515053 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1edee3c-4461-4a0c-bf29-0c41dcdde781 admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:16.515053 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1edee3c-4461-4a0c-bf29-0c41dcdde781 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:03:16.515053 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1edee3c-4461-4a0c-bf29-0c41dcdde781 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:16.515599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1edee3c-4461-4a0c-bf29-0c41dcdde781 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:16.518826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1068/2114] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:16 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:03:16.522790 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674515a4-50f5-45c0-a964-420bddeabbfb admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:16.535852 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674515a4-50f5-45c0-a964-420bddeabbfb admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:16.542126 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1047/2115] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:03:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 14:03:16.549727 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1edee3c-4461-4a0c-bf29-0c41dcdde781 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:16.552855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629ca533-7f34-404a-bc18-942173e7cc81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:16.553470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629ca533-7f34-404a-bc18-942173e7cc81 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:16.553748 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629ca533-7f34-404a-bc18-942173e7cc81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:16.554027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629ca533-7f34-404a-bc18-942173e7cc81 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:16.554706 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629ca533-7f34-404a-bc18-942173e7cc81 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:16.556868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629ca533-7f34-404a-bc18-942173e7cc81 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:16.566232 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1edee3c-4461-4a0c-bf29-0c41dcdde781 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:16.570009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0055fcff-8e5c-4ce2-8bd8-a4d73af6a574 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:16.570880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0055fcff-8e5c-4ce2-8bd8-a4d73af6a574 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:16.571160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0055fcff-8e5c-4ce2-8bd8-a4d73af6a574 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:16.571550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0055fcff-8e5c-4ce2-8bd8-a4d73af6a574 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:16.572246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0055fcff-8e5c-4ce2-8bd8-a4d73af6a574 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:16.573629 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1069/2116] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:16 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:16.574550 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629ca533-7f34-404a-bc18-942173e7cc81 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:16.574787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0055fcff-8e5c-4ce2-8bd8-a4d73af6a574 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:16.585436 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629ca533-7f34-404a-bc18-942173e7cc81 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:16.590963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f59d05-5bbf-47a8-9211-940fbcf740bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:16.591630 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f59d05-5bbf-47a8-9211-940fbcf740bd admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:16.592282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f59d05-5bbf-47a8-9211-940fbcf740bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:16.592282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f59d05-5bbf-47a8-9211-940fbcf740bd admin admin] PATH_INFO: `/v3/projects/901178273d85449e9cdaa5de235fb3e9/users/9359e90c4861470d9cc060abb9c3b203/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:16.608016 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f59d05-5bbf-47a8-9211-940fbcf740bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=901178273d85449e9cdaa5de235fb3e9, user_id=9359e90c4861470d9cc060abb9c3b203, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:16.611588 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f59d05-5bbf-47a8-9211-940fbcf740bd admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:16.669561 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1048/2117] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:16 2022] POST /identity/v3/projects => generated 373 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:16.675052 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0055fcff-8e5c-4ce2-8bd8-a4d73af6a574 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:16.684443 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1070/2118] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:16 2022] PUT /identity/v3/projects/901178273d85449e9cdaa5de235fb3e9/users/9359e90c4861470d9cc060abb9c3b203/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:16.690728 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a328d601-ac2d-4a65-a681-71ce3c2cf0d8 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:16.690949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a328d601-ac2d-4a65-a681-71ce3c2cf0d8 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:16.691225 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a328d601-ac2d-4a65-a681-71ce3c2cf0d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:16.710245 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-a328d601-ac2d-4a65-a681-71ce3c2cf0d8 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:16.711541 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a328d601-ac2d-4a65-a681-71ce3c2cf0d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:16.739666 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0055fcff-8e5c-4ce2-8bd8-a4d73af6a574 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:16.775540 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1071/2119] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:16.787109 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a328d601-ac2d-4a65-a681-71ce3c2cf0d8 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:16.788644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a328d601-ac2d-4a65-a681-71ce3c2cf0d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:16.806324 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ce3f36-8b8a-4b29-b4a1-39780278b7f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:16.807108 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ce3f36-8b8a-4b29-b4a1-39780278b7f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:16.807463 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ce3f36-8b8a-4b29-b4a1-39780278b7f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:16.807808 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ce3f36-8b8a-4b29-b4a1-39780278b7f8 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:16.808559 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ce3f36-8b8a-4b29-b4a1-39780278b7f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:16.810617 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ce3f36-8b8a-4b29-b4a1-39780278b7f8 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:16.848676 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a328d601-ac2d-4a65-a681-71ce3c2cf0d8 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:16.852292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514e4cd3-41a5-444d-8a16-1852f36f4169 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:16.852980 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514e4cd3-41a5-444d-8a16-1852f36f4169 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:16.853259 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514e4cd3-41a5-444d-8a16-1852f36f4169 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:16.853537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514e4cd3-41a5-444d-8a16-1852f36f4169 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:16.862895 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514e4cd3-41a5-444d-8a16-1852f36f4169 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:16.864313 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514e4cd3-41a5-444d-8a16-1852f36f4169 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:16.873615 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1072/2120] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:16.908544 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1049/2121] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:16 2022] POST /identity/v3/users => generated 287 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:16.912958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514e4cd3-41a5-444d-8a16-1852f36f4169 service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:16.926270 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514e4cd3-41a5-444d-8a16-1852f36f4169 service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:16.933047 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c744cff-211c-484a-9e91-3c4b93444c0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:16.933960 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c744cff-211c-484a-9e91-3c4b93444c0e admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:16.934383 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c744cff-211c-484a-9e91-3c4b93444c0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:16.934755 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c744cff-211c-484a-9e91-3c4b93444c0e admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:16.935673 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c744cff-211c-484a-9e91-3c4b93444c0e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:16.937353 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c744cff-211c-484a-9e91-3c4b93444c0e admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:16.948336 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1073/2122] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:16 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:16.952363 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ce3f36-8b8a-4b29-b4a1-39780278b7f8 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:16.963548 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ce3f36-8b8a-4b29-b4a1-39780278b7f8 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:16.969072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12f1df8-4519-475d-a0fe-41eb79cae58d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:16.970974 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12f1df8-4519-475d-a0fe-41eb79cae58d admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:16.971261 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12f1df8-4519-475d-a0fe-41eb79cae58d admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:16.971537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12f1df8-4519-475d-a0fe-41eb79cae58d admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:16.972268 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12f1df8-4519-475d-a0fe-41eb79cae58d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:16.974001 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12f1df8-4519-475d-a0fe-41eb79cae58d admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:16.984178 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1050/2123] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:16 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:16.988437 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c744cff-211c-484a-9e91-3c4b93444c0e admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:16.999299 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c744cff-211c-484a-9e91-3c4b93444c0e admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:17.007079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd326a23-6e19-43a9-91cb-e5a858bb0a58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:17.008085 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd326a23-6e19-43a9-91cb-e5a858bb0a58 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:17.008510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd326a23-6e19-43a9-91cb-e5a858bb0a58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:17.008949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd326a23-6e19-43a9-91cb-e5a858bb0a58 admin admin] PATH_INFO: `/v3/projects/48d448100e564f57b63bf842f9696bff/users/ba44576cd51d4e9e9e3f47e078d9ced4/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:17.022648 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:17.028518 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd326a23-6e19-43a9-91cb-e5a858bb0a58 admin admin] RBAC: Authorizing `identity:create_grant(project_id=48d448100e564f57b63bf842f9696bff, user_id=ba44576cd51d4e9e9e3f47e078d9ced4, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:17.032775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd326a23-6e19-43a9-91cb-e5a858bb0a58 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:17.096584 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:17.107795 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1074/2124] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:16 2022] PUT /identity/v3/projects/48d448100e564f57b63bf842f9696bff/users/ba44576cd51d4e9e9e3f47e078d9ced4/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:17.118950 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c12988e-2fa8-413a-86d2-29645706b5bc None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:17.119906 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c12988e-2fa8-413a-86d2-29645706b5bc None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:17.119906 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c12988e-2fa8-413a-86d2-29645706b5bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:17.149919 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-9c12988e-2fa8-413a-86d2-29645706b5bc None None] MFA Rules not processed for user `ba44576cd51d4e9e9e3f47e078d9ced4`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:17.151662 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9c12988e-2fa8-413a-86d2-29645706b5bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:17.248783 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1051/2125] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:17 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:17.262664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd326a23-6e19-43a9-91cb-e5a858bb0a58 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:17.263970 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-cd326a23-6e19-43a9-91cb-e5a858bb0a58 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:17.333882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd326a23-6e19-43a9-91cb-e5a858bb0a58 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:17.338414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea053b2-ebd5-43be-8e63-f64a13247a33 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:17.339074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea053b2-ebd5-43be-8e63-f64a13247a33 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:17.339382 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea053b2-ebd5-43be-8e63-f64a13247a33 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:17.339790 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea053b2-ebd5-43be-8e63-f64a13247a33 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:17.352815 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea053b2-ebd5-43be-8e63-f64a13247a33 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:17.354562 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea053b2-ebd5-43be-8e63-f64a13247a33 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:17.365426 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1075/2126] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:17 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:03:17.388483 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c12988e-2fa8-413a-86d2-29645706b5bc None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:17.459883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c12988e-2fa8-413a-86d2-29645706b5bc None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:17.465440 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fd89a9-a888-4e85-9308-eaf7628d12fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:17.466190 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fd89a9-a888-4e85-9308-eaf7628d12fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:17.466498 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fd89a9-a888-4e85-9308-eaf7628d12fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:17.466776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fd89a9-a888-4e85-9308-eaf7628d12fd admin admin] PATH_INFO: `/v3/users/ba44576cd51d4e9e9e3f47e078d9ced4` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:17.468566 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fd89a9-a888-4e85-9308-eaf7628d12fd admin admin] RBAC: Authorizing `identity:delete_user(user_id=ba44576cd51d4e9e9e3f47e078d9ced4)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:17.470597 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fd89a9-a888-4e85-9308-eaf7628d12fd admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:17.714975 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-81fd89a9-a888-4e85-9308-eaf7628d12fd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ba44576cd51d4e9e9e3f47e078d9ced4'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:17.821984 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-81fd89a9-a888-4e85-9308-eaf7628d12fd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ba44576cd51d4e9e9e3f47e078d9ced4'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:17.822719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-81fd89a9-a888-4e85-9308-eaf7628d12fd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ba44576cd51d4e9e9e3f47e078d9ced4'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:17.838734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-81fd89a9-a888-4e85-9308-eaf7628d12fd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ba44576cd51d4e9e9e3f47e078d9ced4'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:17.847946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-81fd89a9-a888-4e85-9308-eaf7628d12fd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ba44576cd51d4e9e9e3f47e078d9ced4'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:17.858106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1052/2127] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:17 2022] DELETE /identity/v3/users/ba44576cd51d4e9e9e3f47e078d9ced4 => generated 0 bytes in 470 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:17.863076 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385ec830-d3d1-4aa4-94c8-6547a8736768 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:17.864139 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385ec830-d3d1-4aa4-94c8-6547a8736768 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:17.864587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385ec830-d3d1-4aa4-94c8-6547a8736768 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:17.886529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-385ec830-d3d1-4aa4-94c8-6547a8736768 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:17.887977 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-385ec830-d3d1-4aa4-94c8-6547a8736768 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:17.970356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:17.988175 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1076/2128] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:17.995080 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fd89a9-a888-4e85-9308-eaf7628d12fd admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:17.996717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-81fd89a9-a888-4e85-9308-eaf7628d12fd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:18.029619 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:18.063622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fd89a9-a888-4e85-9308-eaf7628d12fd admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:18.067733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccc0ee5-7b3f-4424-830f-7adf5c0c1629 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:18.068452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccc0ee5-7b3f-4424-830f-7adf5c0c1629 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:18.068823 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccc0ee5-7b3f-4424-830f-7adf5c0c1629 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:18.069224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccc0ee5-7b3f-4424-830f-7adf5c0c1629 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:18.081890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccc0ee5-7b3f-4424-830f-7adf5c0c1629 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:18.083577 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccc0ee5-7b3f-4424-830f-7adf5c0c1629 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:18.094426 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1053/2129] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:18.507349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:18.559299 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:18.598260 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385ec830-d3d1-4aa4-94c8-6547a8736768 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:18.612300 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-385ec830-d3d1-4aa4-94c8-6547a8736768 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:18.643599 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385ec830-d3d1-4aa4-94c8-6547a8736768 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:18.649021 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81469abf-ff2b-4e4f-a1d7-701b5eb242cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:18.649969 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81469abf-ff2b-4e4f-a1d7-701b5eb242cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:18.650322 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81469abf-ff2b-4e4f-a1d7-701b5eb242cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:18.650759 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81469abf-ff2b-4e4f-a1d7-701b5eb242cd admin admin] PATH_INFO: `/v3/projects/48d448100e564f57b63bf842f9696bff` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:18.653012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81469abf-ff2b-4e4f-a1d7-701b5eb242cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=48d448100e564f57b63bf842f9696bff)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:18.655656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81469abf-ff2b-4e4f-a1d7-701b5eb242cd admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:18.805385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-81469abf-ff2b-4e4f-a1d7-701b5eb242cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '48d448100e564f57b63bf842f9696bff', 'request_id': 'req-81469abf-ff2b-4e4f-a1d7-701b5eb242cd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:18.948553 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-81469abf-ff2b-4e4f-a1d7-701b5eb242cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '48d448100e564f57b63bf842f9696bff', 'request_id': 'req-81469abf-ff2b-4e4f-a1d7-701b5eb242cd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:18.959540 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-81469abf-ff2b-4e4f-a1d7-701b5eb242cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '48d448100e564f57b63bf842f9696bff', 'request_id': 'req-81469abf-ff2b-4e4f-a1d7-701b5eb242cd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:18.982581 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-81469abf-ff2b-4e4f-a1d7-701b5eb242cd admin admin] The token cache is being invalidate because project 48d448100e564f57b63bf842f9696bff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:18.982927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-81469abf-ff2b-4e4f-a1d7-701b5eb242cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:18.983584 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-81469abf-ff2b-4e4f-a1d7-701b5eb242cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:18.985191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1077/2130] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:18 2022] DELETE /identity/v3/projects/48d448100e564f57b63bf842f9696bff => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:18.990443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28f3445-2761-49ea-9b14-feeb49943c00 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:18.991020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28f3445-2761-49ea-9b14-feeb49943c00 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:18.991290 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28f3445-2761-49ea-9b14-feeb49943c00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:19.014008 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-d28f3445-2761-49ea-9b14-feeb49943c00 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:19.015293 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d28f3445-2761-49ea-9b14-feeb49943c00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:19.080683 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81469abf-ff2b-4e4f-a1d7-701b5eb242cd admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:19.082168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-81469abf-ff2b-4e4f-a1d7-701b5eb242cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:19.171579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1054/2131] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:19.180555 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e78e2dc-926e-46b3-a1ab-85cad6f07103 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:19.181284 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e78e2dc-926e-46b3-a1ab-85cad6f07103 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:19.181786 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e78e2dc-926e-46b3-a1ab-85cad6f07103 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:19.182241 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81469abf-ff2b-4e4f-a1d7-701b5eb242cd admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:19.188581 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af44c144-ca24-45a0-bd6c-de79088d77e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:19.189310 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af44c144-ca24-45a0-bd6c-de79088d77e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:19.189597 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af44c144-ca24-45a0-bd6c-de79088d77e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:19.189873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af44c144-ca24-45a0-bd6c-de79088d77e7 admin admin] PATH_INFO: `/v3/users/5fd44db8498f47c2910a623fe7fb6f26` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:19.191069 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af44c144-ca24-45a0-bd6c-de79088d77e7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5fd44db8498f47c2910a623fe7fb6f26)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:19.193024 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af44c144-ca24-45a0-bd6c-de79088d77e7 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:19.211010 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-5e78e2dc-926e-46b3-a1ab-85cad6f07103 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:19.212301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-5e78e2dc-926e-46b3-a1ab-85cad6f07103 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:19.234680 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1055/2132] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:19.234680 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e78e2dc-926e-46b3-a1ab-85cad6f07103 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:19.241956 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e78e2dc-926e-46b3-a1ab-85cad6f07103 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:19.252134 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b631d5-d5bd-4c35-be01-5103e0e8ef4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:19.253040 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b631d5-d5bd-4c35-be01-5103e0e8ef4c admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:19.253511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b631d5-d5bd-4c35-be01-5103e0e8ef4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:19.253794 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b631d5-d5bd-4c35-be01-5103e0e8ef4c admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:19.254496 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b631d5-d5bd-4c35-be01-5103e0e8ef4c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:03:19.254869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b631d5-d5bd-4c35-be01-5103e0e8ef4c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:19.256671 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b631d5-d5bd-4c35-be01-5103e0e8ef4c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:19.282266 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1056/2133] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:03:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:03:19.284297 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b631d5-d5bd-4c35-be01-5103e0e8ef4c admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:19.295713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b631d5-d5bd-4c35-be01-5103e0e8ef4c admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:19.301869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0573ac-196f-4f28-a9bf-47dae27e9922 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:19.302506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0573ac-196f-4f28-a9bf-47dae27e9922 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:19.302786 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0573ac-196f-4f28-a9bf-47dae27e9922 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:19.303053 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0573ac-196f-4f28-a9bf-47dae27e9922 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:19.303719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0573ac-196f-4f28-a9bf-47dae27e9922 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:19.305955 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0573ac-196f-4f28-a9bf-47dae27e9922 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:19.424410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-af44c144-ca24-45a0-bd6c-de79088d77e7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5fd44db8498f47c2910a623fe7fb6f26'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:19.445959 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1057/2134] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:19 2022] POST /identity/v3/projects => generated 377 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:03:19.450012 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0573ac-196f-4f28-a9bf-47dae27e9922 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:19.495327 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-af44c144-ca24-45a0-bd6c-de79088d77e7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5fd44db8498f47c2910a623fe7fb6f26'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:19.513030 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-af44c144-ca24-45a0-bd6c-de79088d77e7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5fd44db8498f47c2910a623fe7fb6f26'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:19.514405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-af44c144-ca24-45a0-bd6c-de79088d77e7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5fd44db8498f47c2910a623fe7fb6f26'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:19.524696 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-af44c144-ca24-45a0-bd6c-de79088d77e7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5fd44db8498f47c2910a623fe7fb6f26'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:19.531153 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1078/2135] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:19 2022] DELETE /identity/v3/users/5fd44db8498f47c2910a623fe7fb6f26 => generated 0 bytes in 451 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:19.534686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0573ac-196f-4f28-a9bf-47dae27e9922 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:19.535943 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04883fd8-ead5-49c5-9955-2475037b549e None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:19.536228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04883fd8-ead5-49c5-9955-2475037b549e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:19.536503 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04883fd8-ead5-49c5-9955-2475037b549e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:19.565981 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-04883fd8-ead5-49c5-9955-2475037b549e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:19.567401 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-04883fd8-ead5-49c5-9955-2475037b549e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:19.618108 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dfac40-6b66-4281-9ae8-58216cccad2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:19.618880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dfac40-6b66-4281-9ae8-58216cccad2d admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:19.619258 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dfac40-6b66-4281-9ae8-58216cccad2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:19.619636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9dfac40-6b66-4281-9ae8-58216cccad2d admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:19.621215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9dfac40-6b66-4281-9ae8-58216cccad2d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:19.637603 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9dfac40-6b66-4281-9ae8-58216cccad2d admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:19.690778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1079/2136] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:19.701009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04883fd8-ead5-49c5-9955-2475037b549e None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:19.707350 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-04883fd8-ead5-49c5-9955-2475037b549e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:19.748272 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1058/2137] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:19 2022] POST /identity/v3/users => generated 289 bytes in 296 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:19.749797 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dfac40-6b66-4281-9ae8-58216cccad2d admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:19.751965 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-b9dfac40-6b66-4281-9ae8-58216cccad2d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:19.767023 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9dfac40-6b66-4281-9ae8-58216cccad2d admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:19.773507 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b5f83e-f66a-4ede-8f90-81a93d87096b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:19.774250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b5f83e-f66a-4ede-8f90-81a93d87096b admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:19.774567 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b5f83e-f66a-4ede-8f90-81a93d87096b admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:19.774887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b5f83e-f66a-4ede-8f90-81a93d87096b admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:19.775680 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b5f83e-f66a-4ede-8f90-81a93d87096b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:19.779707 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b5f83e-f66a-4ede-8f90-81a93d87096b admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:19.804966 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1059/2138] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:19 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:03:19.806014 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b5f83e-f66a-4ede-8f90-81a93d87096b admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:19.823044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b5f83e-f66a-4ede-8f90-81a93d87096b admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:19.827896 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04883fd8-ead5-49c5-9955-2475037b549e None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:19.828400 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ef5fed-3e0f-4d11-aa3a-0698a41d563c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:19.829946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ef5fed-3e0f-4d11-aa3a-0698a41d563c admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:19.830227 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ef5fed-3e0f-4d11-aa3a-0698a41d563c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:19.830531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ef5fed-3e0f-4d11-aa3a-0698a41d563c admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:19.831321 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ef5fed-3e0f-4d11-aa3a-0698a41d563c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:19.832777 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55a907a-1c5d-44ff-b823-e34c1bc8b2dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:19.832912 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ef5fed-3e0f-4d11-aa3a-0698a41d563c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:19.833386 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55a907a-1c5d-44ff-b823-e34c1bc8b2dd service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:19.833717 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55a907a-1c5d-44ff-b823-e34c1bc8b2dd service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:19.833985 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55a907a-1c5d-44ff-b823-e34c1bc8b2dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:19.846936 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1060/2139] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:19 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:03:19.850914 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55a907a-1c5d-44ff-b823-e34c1bc8b2dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:19.851688 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ef5fed-3e0f-4d11-aa3a-0698a41d563c admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:19.852540 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55a907a-1c5d-44ff-b823-e34c1bc8b2dd service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:19.866175 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1080/2140] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:19.866481 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ef5fed-3e0f-4d11-aa3a-0698a41d563c admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:19.871672 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3c00b8-1525-420b-a372-c5af055a9498 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:19.872354 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3c00b8-1525-420b-a372-c5af055a9498 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:19.872682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3c00b8-1525-420b-a372-c5af055a9498 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:19.872890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3c00b8-1525-420b-a372-c5af055a9498 admin admin] PATH_INFO: `/v3/projects/15cacfd8a7114dc891d48c3e30df3733/users/35d18ce141664ccdbd9c0a1aac0af358/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:19.890517 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3c00b8-1525-420b-a372-c5af055a9498 admin admin] RBAC: Authorizing `identity:create_grant(project_id=15cacfd8a7114dc891d48c3e30df3733, user_id=35d18ce141664ccdbd9c0a1aac0af358, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:19.894622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3c00b8-1525-420b-a372-c5af055a9498 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:20.079330 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1061/2141] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:19 2022] PUT /identity/v3/projects/15cacfd8a7114dc891d48c3e30df3733/users/35d18ce141664ccdbd9c0a1aac0af358/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:20.089946 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cebe46-34bd-4990-9a3a-3fd1e1121806 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:20.090836 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cebe46-34bd-4990-9a3a-3fd1e1121806 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:20.091113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cebe46-34bd-4990-9a3a-3fd1e1121806 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:20.128753 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-b7cebe46-34bd-4990-9a3a-3fd1e1121806 None None] MFA Rules not processed for user `35d18ce141664ccdbd9c0a1aac0af358`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:20.132784 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-b7cebe46-34bd-4990-9a3a-3fd1e1121806 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:20.252210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1081/2142] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:20 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:20.267260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3c00b8-1525-420b-a372-c5af055a9498 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:20.269527 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-db3c00b8-1525-420b-a372-c5af055a9498 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:20.390780 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:20.419517 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3c00b8-1525-420b-a372-c5af055a9498 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:20.424416 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c88f1d-45b6-4efc-95e6-48e36c92f165 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'acdf4a7bd608492390176dcc42ca1267', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:20.427905 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c88f1d-45b6-4efc-95e6-48e36c92f165 service glance] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:20.428243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c88f1d-45b6-4efc-95e6-48e36c92f165 service glance] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:20.428523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c88f1d-45b6-4efc-95e6-48e36c92f165 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:20.467080 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c88f1d-45b6-4efc-95e6-48e36c92f165 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:20.468774 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c88f1d-45b6-4efc-95e6-48e36c92f165 service glance] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:20.482138 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1062/2143] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:20 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:20.513081 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:20.558770 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7cebe46-34bd-4990-9a3a-3fd1e1121806 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:20.559974 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-b7cebe46-34bd-4990-9a3a-3fd1e1121806 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:20.680384 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7cebe46-34bd-4990-9a3a-3fd1e1121806 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:20.685161 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd318543-e94c-4f13-ba3d-d8fb334d6839 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:20.685853 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd318543-e94c-4f13-ba3d-d8fb334d6839 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:20.686469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd318543-e94c-4f13-ba3d-d8fb334d6839 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:20.686469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd318543-e94c-4f13-ba3d-d8fb334d6839 admin admin] PATH_INFO: `/v3/projects/ca45b248032f449da726c9dd9bba62a0` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:20.687506 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd318543-e94c-4f13-ba3d-d8fb334d6839 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ca45b248032f449da726c9dd9bba62a0)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:20.690142 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd318543-e94c-4f13-ba3d-d8fb334d6839 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:20.842401 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-dd318543-e94c-4f13-ba3d-d8fb334d6839 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ca45b248032f449da726c9dd9bba62a0', 'request_id': 'req-dd318543-e94c-4f13-ba3d-d8fb334d6839'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:20.953404 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-dd318543-e94c-4f13-ba3d-d8fb334d6839 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ca45b248032f449da726c9dd9bba62a0', 'request_id': 'req-dd318543-e94c-4f13-ba3d-d8fb334d6839'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:20.961399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-dd318543-e94c-4f13-ba3d-d8fb334d6839 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ca45b248032f449da726c9dd9bba62a0', 'request_id': 'req-dd318543-e94c-4f13-ba3d-d8fb334d6839'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:20.980774 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-dd318543-e94c-4f13-ba3d-d8fb334d6839 admin admin] The token cache is being invalidate because project ca45b248032f449da726c9dd9bba62a0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:20.981552 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-dd318543-e94c-4f13-ba3d-d8fb334d6839 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:20.981911 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-dd318543-e94c-4f13-ba3d-d8fb334d6839 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:20.983808 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1082/2144] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:20 2022] DELETE /identity/v3/projects/ca45b248032f449da726c9dd9bba62a0 => generated 0 bytes in 427 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:20.989924 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c88f1d-45b6-4efc-95e6-48e36c92f165 service glance] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:21.001584 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c88f1d-45b6-4efc-95e6-48e36c92f165 service glance] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:21.138359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c057dd9-6f00-4062-b442-c9608ea44d2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:21.140743 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c057dd9-6f00-4062-b442-c9608ea44d2e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:21.143671 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c057dd9-6f00-4062-b442-c9608ea44d2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:21.143671 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c057dd9-6f00-4062-b442-c9608ea44d2e admin admin] PATH_INFO: `/v3/users/ab51b4b0402c4e07b9e8459b8479d25a` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:21.145752 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c057dd9-6f00-4062-b442-c9608ea44d2e admin admin] RBAC: Authorizing `identity:delete_user(user_id=ab51b4b0402c4e07b9e8459b8479d25a)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:21.148691 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c057dd9-6f00-4062-b442-c9608ea44d2e admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:21.408002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:21.427152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3c057dd9-6f00-4062-b442-c9608ea44d2e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ab51b4b0402c4e07b9e8459b8479d25a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:21.541194 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3c057dd9-6f00-4062-b442-c9608ea44d2e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ab51b4b0402c4e07b9e8459b8479d25a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:21.541898 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3c057dd9-6f00-4062-b442-c9608ea44d2e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ab51b4b0402c4e07b9e8459b8479d25a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:21.579379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3c057dd9-6f00-4062-b442-c9608ea44d2e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ab51b4b0402c4e07b9e8459b8479d25a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:21.605088 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3c057dd9-6f00-4062-b442-c9608ea44d2e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ab51b4b0402c4e07b9e8459b8479d25a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:21.627446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1063/2145] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:20 2022] DELETE /identity/v3/users/ab51b4b0402c4e07b9e8459b8479d25a => generated 0 bytes in 639 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:21.936861 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd318543-e94c-4f13-ba3d-d8fb334d6839 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:21.939394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-dd318543-e94c-4f13-ba3d-d8fb334d6839 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:21.952693 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd318543-e94c-4f13-ba3d-d8fb334d6839 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:22.087025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43b1b82-af32-4748-a9b9-9a35a6534bdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:22.087763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43b1b82-af32-4748-a9b9-9a35a6534bdc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:22.088165 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43b1b82-af32-4748-a9b9-9a35a6534bdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:22.088505 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43b1b82-af32-4748-a9b9-9a35a6534bdc admin admin] PATH_INFO: `/v3/users/429548cbe4544ea2b2632bf17bb4bd5a` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:22.091092 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43b1b82-af32-4748-a9b9-9a35a6534bdc admin admin] RBAC: Authorizing `identity:delete_user(user_id=429548cbe4544ea2b2632bf17bb4bd5a)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:22.093099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43b1b82-af32-4748-a9b9-9a35a6534bdc admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:22.197762 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c057dd9-6f00-4062-b442-c9608ea44d2e admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:22.200917 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3c057dd9-6f00-4062-b442-c9608ea44d2e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:22.224490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c057dd9-6f00-4062-b442-c9608ea44d2e admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:22.230952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c52041-c35c-4038-a3b1-50d90030a2bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:22.231673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c52041-c35c-4038-a3b1-50d90030a2bd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:22.232089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c52041-c35c-4038-a3b1-50d90030a2bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:22.232922 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c52041-c35c-4038-a3b1-50d90030a2bd admin admin] PATH_INFO: `/v3/users/063aa7c6b04c4ee99f9f5f20a872de9a` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:22.234251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c52041-c35c-4038-a3b1-50d90030a2bd admin admin] RBAC: Authorizing `identity:delete_user(user_id=063aa7c6b04c4ee99f9f5f20a872de9a)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:22.238154 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c52041-c35c-4038-a3b1-50d90030a2bd admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:22.370362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f43b1b82-af32-4748-a9b9-9a35a6534bdc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '429548cbe4544ea2b2632bf17bb4bd5a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:22.450051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f43b1b82-af32-4748-a9b9-9a35a6534bdc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '429548cbe4544ea2b2632bf17bb4bd5a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:22.477833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f43b1b82-af32-4748-a9b9-9a35a6534bdc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '429548cbe4544ea2b2632bf17bb4bd5a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:22.478637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f43b1b82-af32-4748-a9b9-9a35a6534bdc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '429548cbe4544ea2b2632bf17bb4bd5a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:22.479985 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a2c52041-c35c-4038-a3b1-50d90030a2bd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '063aa7c6b04c4ee99f9f5f20a872de9a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:22.496893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f43b1b82-af32-4748-a9b9-9a35a6534bdc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '429548cbe4544ea2b2632bf17bb4bd5a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:22.507900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1083/2146] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:21 2022] DELETE /identity/v3/users/429548cbe4544ea2b2632bf17bb4bd5a => generated 0 bytes in 571 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:22.508873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43b1b82-af32-4748-a9b9-9a35a6534bdc admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:22.510440 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f43b1b82-af32-4748-a9b9-9a35a6534bdc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:22.565010 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a2c52041-c35c-4038-a3b1-50d90030a2bd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '063aa7c6b04c4ee99f9f5f20a872de9a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:22.566897 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a2c52041-c35c-4038-a3b1-50d90030a2bd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '063aa7c6b04c4ee99f9f5f20a872de9a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:22.583265 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a2c52041-c35c-4038-a3b1-50d90030a2bd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '063aa7c6b04c4ee99f9f5f20a872de9a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:22.595281 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a2c52041-c35c-4038-a3b1-50d90030a2bd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '063aa7c6b04c4ee99f9f5f20a872de9a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:22.608967 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1064/2147] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:22 2022] DELETE /identity/v3/users/063aa7c6b04c4ee99f9f5f20a872de9a => generated 0 bytes in 407 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:22.611370 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8650d2b9-88f5-4eaf-b855-87ead8d3a3f3 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:22.612010 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8650d2b9-88f5-4eaf-b855-87ead8d3a3f3 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:22.612510 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8650d2b9-88f5-4eaf-b855-87ead8d3a3f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:22.657852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-8650d2b9-88f5-4eaf-b855-87ead8d3a3f3 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:22.659814 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8650d2b9-88f5-4eaf-b855-87ead8d3a3f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:22.660703 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43b1b82-af32-4748-a9b9-9a35a6534bdc admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:22.719856 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:22.758635 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6a9bb3-fb2e-4bdc-ae4d-3ea38cbb1833 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:22.758635 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6a9bb3-fb2e-4bdc-ae4d-3ea38cbb1833 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:22.758635 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6a9bb3-fb2e-4bdc-ae4d-3ea38cbb1833 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:22.758635 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb6a9bb3-fb2e-4bdc-ae4d-3ea38cbb1833 admin admin] PATH_INFO: `/v3/projects/53ec96afe7fa43c79713342da619af04` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:22.758959 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6a9bb3-fb2e-4bdc-ae4d-3ea38cbb1833 admin admin] RBAC: Authorizing `identity:delete_project(project_id=53ec96afe7fa43c79713342da619af04)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:22.760415 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb6a9bb3-fb2e-4bdc-ae4d-3ea38cbb1833 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:22.786146 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1065/2148] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:22.789236 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2d237f-9e0c-471f-b5ee-7237355e5a30 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:22.792612 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2d237f-9e0c-471f-b5ee-7237355e5a30 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:22.796539 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2d237f-9e0c-471f-b5ee-7237355e5a30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:22.870644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-bc2d237f-9e0c-471f-b5ee-7237355e5a30 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:22.872963 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-bc2d237f-9e0c-471f-b5ee-7237355e5a30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:22.882832 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1066/2149] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:22.884063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2d237f-9e0c-471f-b5ee-7237355e5a30 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:22.892437 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-bc2d237f-9e0c-471f-b5ee-7237355e5a30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:22.961106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-bb6a9bb3-fb2e-4bdc-ae4d-3ea38cbb1833 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '53ec96afe7fa43c79713342da619af04', 'request_id': 'req-bb6a9bb3-fb2e-4bdc-ae4d-3ea38cbb1833'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:23.022038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2d237f-9e0c-471f-b5ee-7237355e5a30 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:23.049982 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-bb6a9bb3-fb2e-4bdc-ae4d-3ea38cbb1833 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '53ec96afe7fa43c79713342da619af04', 'request_id': 'req-bb6a9bb3-fb2e-4bdc-ae4d-3ea38cbb1833'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:23.057512 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-bb6a9bb3-fb2e-4bdc-ae4d-3ea38cbb1833 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '53ec96afe7fa43c79713342da619af04', 'request_id': 'req-bb6a9bb3-fb2e-4bdc-ae4d-3ea38cbb1833'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:23.087992 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-bb6a9bb3-fb2e-4bdc-ae4d-3ea38cbb1833 admin admin] The token cache is being invalidate because project 53ec96afe7fa43c79713342da619af04 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:23.088511 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-bb6a9bb3-fb2e-4bdc-ae4d-3ea38cbb1833 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:23.095121 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-bb6a9bb3-fb2e-4bdc-ae4d-3ea38cbb1833 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:23.103721 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1084/2150] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:22 2022] DELETE /identity/v3/projects/53ec96afe7fa43c79713342da619af04 => generated 0 bytes in 595 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:23.107307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6a9bb3-fb2e-4bdc-ae4d-3ea38cbb1833 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:23.108760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-bb6a9bb3-fb2e-4bdc-ae4d-3ea38cbb1833 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:23.158284 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533a9d5a-b151-4b4b-918c-2fd78f0f2d05 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:23.158973 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533a9d5a-b151-4b4b-918c-2fd78f0f2d05 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:23.159299 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533a9d5a-b151-4b4b-918c-2fd78f0f2d05 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:23.159589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533a9d5a-b151-4b4b-918c-2fd78f0f2d05 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:23.163223 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-533a9d5a-b151-4b4b-918c-2fd78f0f2d05 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:23.260516 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb6a9bb3-fb2e-4bdc-ae4d-3ea38cbb1833 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:23.267315 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d6f028-27fb-4b7f-a8e9-67f060319c18 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:23.268183 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d6f028-27fb-4b7f-a8e9-67f060319c18 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:23.268637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d6f028-27fb-4b7f-a8e9-67f060319c18 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:23.269148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d6f028-27fb-4b7f-a8e9-67f060319c18 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:23.271584 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-65d6f028-27fb-4b7f-a8e9-67f060319c18 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:23.313635 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533a9d5a-b151-4b4b-918c-2fd78f0f2d05 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:23.313635 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533a9d5a-b151-4b4b-918c-2fd78f0f2d05 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:23.328922 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1067/2151] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:23.331784 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93010afa-47ba-428c-8ad1-020c6045e10c None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:23.332576 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93010afa-47ba-428c-8ad1-020c6045e10c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:23.332984 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93010afa-47ba-428c-8ad1-020c6045e10c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:23.357188 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-93010afa-47ba-428c-8ad1-020c6045e10c None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:23.358633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-93010afa-47ba-428c-8ad1-020c6045e10c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:23.377796 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1068/2152] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:23.386583 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867f7ead-4e9b-4191-8be1-12385aaa66be None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:23.387847 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867f7ead-4e9b-4191-8be1-12385aaa66be None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:23.390522 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867f7ead-4e9b-4191-8be1-12385aaa66be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:23.398891 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d6f028-27fb-4b7f-a8e9-67f060319c18 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:23.400719 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d6f028-27fb-4b7f-a8e9-67f060319c18 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:23.412629 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1085/2153] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:23.462565 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-867f7ead-4e9b-4191-8be1-12385aaa66be None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:23.463616 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-867f7ead-4e9b-4191-8be1-12385aaa66be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:23.478850 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1069/2154] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:23.485728 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d6f028-27fb-4b7f-a8e9-67f060319c18 service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:23.497802 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d6f028-27fb-4b7f-a8e9-67f060319c18 service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:23.504542 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:DELETE=3,keystone:SELECT=93,keystone:INSERT=4 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:23.516526 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83972d0d-0aa3-49c1-9a96-bd90f16842ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:23.517163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83972d0d-0aa3-49c1-9a96-bd90f16842ae admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:23.517443 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83972d0d-0aa3-49c1-9a96-bd90f16842ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:23.517765 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83972d0d-0aa3-49c1-9a96-bd90f16842ae admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:23.518500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83972d0d-0aa3-49c1-9a96-bd90f16842ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:03:23.518836 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83972d0d-0aa3-49c1-9a96-bd90f16842ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:23.520636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83972d0d-0aa3-49c1-9a96-bd90f16842ae admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:23.537028 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1086/2155] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:03:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:03:23.545134 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867f7ead-4e9b-4191-8be1-12385aaa66be None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:23.565826 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867f7ead-4e9b-4191-8be1-12385aaa66be None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:23.578658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5416caa5-5f56-4975-98f9-d15de0c6df76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:23.582890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5416caa5-5f56-4975-98f9-d15de0c6df76 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:23.582890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5416caa5-5f56-4975-98f9-d15de0c6df76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:23.582890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5416caa5-5f56-4975-98f9-d15de0c6df76 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:23.584383 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5416caa5-5f56-4975-98f9-d15de0c6df76 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:23.586537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5416caa5-5f56-4975-98f9-d15de0c6df76 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:23.749038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1070/2156] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:23 2022] POST /identity/v3/projects => generated 375 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:23.751529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83972d0d-0aa3-49c1-9a96-bd90f16842ae admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:23.764476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83972d0d-0aa3-49c1-9a96-bd90f16842ae admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:23.895046 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cab5bc9-a8a5-435c-987a-f7b06685c531 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:23.895740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cab5bc9-a8a5-435c-987a-f7b06685c531 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:23.896077 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cab5bc9-a8a5-435c-987a-f7b06685c531 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:23.896541 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cab5bc9-a8a5-435c-987a-f7b06685c531 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:23.897253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cab5bc9-a8a5-435c-987a-f7b06685c531 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:23.902638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cab5bc9-a8a5-435c-987a-f7b06685c531 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:24.011613 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1087/2157] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:23 2022] POST /identity/v3/users => generated 288 bytes in 260 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:24.016419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5416caa5-5f56-4975-98f9-d15de0c6df76 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:24.030449 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5416caa5-5f56-4975-98f9-d15de0c6df76 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:24.042075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113844ef-86a3-4151-ab1e-f11835c2f8fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:24.042823 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113844ef-86a3-4151-ab1e-f11835c2f8fc admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:24.043284 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113844ef-86a3-4151-ab1e-f11835c2f8fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:24.043610 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113844ef-86a3-4151-ab1e-f11835c2f8fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:24.044341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113844ef-86a3-4151-ab1e-f11835c2f8fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:24.046282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113844ef-86a3-4151-ab1e-f11835c2f8fc admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:24.065165 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1071/2158] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:24 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:24.068676 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cab5bc9-a8a5-435c-987a-f7b06685c531 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:24.080609 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cab5bc9-a8a5-435c-987a-f7b06685c531 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:24.087784 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db2bae1-d99d-49c4-8ffb-fdc8e64bb0cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:24.089907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db2bae1-d99d-49c4-8ffb-fdc8e64bb0cc admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:24.089907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db2bae1-d99d-49c4-8ffb-fdc8e64bb0cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:24.089907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db2bae1-d99d-49c4-8ffb-fdc8e64bb0cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:24.090692 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db2bae1-d99d-49c4-8ffb-fdc8e64bb0cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:24.095256 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db2bae1-d99d-49c4-8ffb-fdc8e64bb0cc admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:24.110263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1088/2159] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:24 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:03:24.117663 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113844ef-86a3-4151-ab1e-f11835c2f8fc admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:24.128183 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113844ef-86a3-4151-ab1e-f11835c2f8fc admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:24.141713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0410152f-daad-410f-abc5-18845c546871 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:24.141713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0410152f-daad-410f-abc5-18845c546871 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:24.141713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0410152f-daad-410f-abc5-18845c546871 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:24.141713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0410152f-daad-410f-abc5-18845c546871 admin admin] PATH_INFO: `/v3/projects/58e31a2ddf7741ac936bd5bd8eb40acb/users/4b7d120c2c0e455b92507aebaf7a31a1/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:24.171763 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0410152f-daad-410f-abc5-18845c546871 admin admin] RBAC: Authorizing `identity:create_grant(project_id=58e31a2ddf7741ac936bd5bd8eb40acb, user_id=4b7d120c2c0e455b92507aebaf7a31a1, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:24.175847 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0410152f-daad-410f-abc5-18845c546871 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:24.176118 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:24.284333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1072/2160] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:24 2022] PUT /identity/v3/projects/58e31a2ddf7741ac936bd5bd8eb40acb/users/4b7d120c2c0e455b92507aebaf7a31a1/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:24.289340 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09df9144-7fe2-4164-8099-e952405bdbde None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:24.289340 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09df9144-7fe2-4164-8099-e952405bdbde None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:24.289340 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09df9144-7fe2-4164-8099-e952405bdbde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:24.340752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-09df9144-7fe2-4164-8099-e952405bdbde None None] MFA Rules not processed for user `4b7d120c2c0e455b92507aebaf7a31a1`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:24.342207 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-09df9144-7fe2-4164-8099-e952405bdbde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:24.375226 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0410152f-daad-410f-abc5-18845c546871 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:24.376973 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-0410152f-daad-410f-abc5-18845c546871 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:24.502925 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1089/2161] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:24 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:24.518609 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09df9144-7fe2-4164-8099-e952405bdbde None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:24.520142 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-09df9144-7fe2-4164-8099-e952405bdbde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:24.532186 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0410152f-daad-410f-abc5-18845c546871 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:24.539251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e18b77-4316-44b5-a889-2c7ba7abf08e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:24.542873 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e18b77-4316-44b5-a889-2c7ba7abf08e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:24.542873 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e18b77-4316-44b5-a889-2c7ba7abf08e admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:24.542873 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e18b77-4316-44b5-a889-2c7ba7abf08e admin admin] PATH_INFO: `/v3/projects/e367eb231b38468c8bd6ebe483d6ecfb` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:24.554103 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e18b77-4316-44b5-a889-2c7ba7abf08e admin admin] RBAC: Authorizing `identity:delete_project(project_id=e367eb231b38468c8bd6ebe483d6ecfb)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:24.564732 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e18b77-4316-44b5-a889-2c7ba7abf08e admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:24.597774 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:24.617208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:24.634786 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09df9144-7fe2-4164-8099-e952405bdbde None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:24.638783 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4bb607-bcad-4c9e-854d-6e5fd2fadbde service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:24.639392 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4bb607-bcad-4c9e-854d-6e5fd2fadbde service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:24.639721 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4bb607-bcad-4c9e-854d-6e5fd2fadbde service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:24.640172 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4bb607-bcad-4c9e-854d-6e5fd2fadbde service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:24.650822 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4bb607-bcad-4c9e-854d-6e5fd2fadbde service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:24.652694 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c4bb607-bcad-4c9e-854d-6e5fd2fadbde service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:24.665340 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1090/2162] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:24 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:03:24.717970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-06e18b77-4316-44b5-a889-2c7ba7abf08e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e367eb231b38468c8bd6ebe483d6ecfb', 'request_id': 'req-06e18b77-4316-44b5-a889-2c7ba7abf08e'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:24.841103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ac74ce-7893-4bb2-a4ee-bd99ece6dfb7 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:24.841391 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ac74ce-7893-4bb2-a4ee-bd99ece6dfb7 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:24.841666 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ac74ce-7893-4bb2-a4ee-bd99ece6dfb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:24.843854 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-06e18b77-4316-44b5-a889-2c7ba7abf08e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e367eb231b38468c8bd6ebe483d6ecfb', 'request_id': 'req-06e18b77-4316-44b5-a889-2c7ba7abf08e'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:24.852906 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-06e18b77-4316-44b5-a889-2c7ba7abf08e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e367eb231b38468c8bd6ebe483d6ecfb', 'request_id': 'req-06e18b77-4316-44b5-a889-2c7ba7abf08e'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:24.878788 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-06e18b77-4316-44b5-a889-2c7ba7abf08e admin admin] The token cache is being invalidate because project e367eb231b38468c8bd6ebe483d6ecfb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:24.878914 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-06e18b77-4316-44b5-a889-2c7ba7abf08e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:24.884230 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-06e18b77-4316-44b5-a889-2c7ba7abf08e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:24.884230 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1073/2163] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:24 2022] DELETE /identity/v3/projects/e367eb231b38468c8bd6ebe483d6ecfb => generated 0 bytes in 510 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:24.885495 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e18b77-4316-44b5-a889-2c7ba7abf08e admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:24.885977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-06e18b77-4316-44b5-a889-2c7ba7abf08e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:24.899204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-66ac74ce-7893-4bb2-a4ee-bd99ece6dfb7 None None] MFA Rules not processed for user `9359e90c4861470d9cc060abb9c3b203`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:24.900435 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-66ac74ce-7893-4bb2-a4ee-bd99ece6dfb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:25.004276 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e18b77-4316-44b5-a889-2c7ba7abf08e admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:25.008112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1091/2164] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:24 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:25.009208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ac74ce-7893-4bb2-a4ee-bd99ece6dfb7 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:25.021498 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-66ac74ce-7893-4bb2-a4ee-bd99ece6dfb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:25.039303 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21b2d02-a74d-403b-8576-774295798ccc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:25.039964 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21b2d02-a74d-403b-8576-774295798ccc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:25.044219 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21b2d02-a74d-403b-8576-774295798ccc admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:25.044498 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21b2d02-a74d-403b-8576-774295798ccc admin admin] PATH_INFO: `/v3/projects/b5f5f78825d5490bbe170255f4c85070` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:25.046163 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21b2d02-a74d-403b-8576-774295798ccc admin admin] RBAC: Authorizing `identity:delete_project(project_id=b5f5f78825d5490bbe170255f4c85070)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:25.048727 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21b2d02-a74d-403b-8576-774295798ccc admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:25.059348 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ac74ce-7893-4bb2-a4ee-bd99ece6dfb7 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:25.067181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcad1ca-78b5-43b8-bfa0-3d6144ae6b52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:25.068308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcad1ca-78b5-43b8-bfa0-3d6144ae6b52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:25.068847 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcad1ca-78b5-43b8-bfa0-3d6144ae6b52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:25.069264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcad1ca-78b5-43b8-bfa0-3d6144ae6b52 admin admin] PATH_INFO: `/v3/users/ceddaa716eb94e2cbcca0eaa1d6096fc` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:25.071663 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcad1ca-78b5-43b8-bfa0-3d6144ae6b52 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ceddaa716eb94e2cbcca0eaa1d6096fc)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:25.081068 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcad1ca-78b5-43b8-bfa0-3d6144ae6b52 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:25.089851 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:25.307426 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d21b2d02-a74d-403b-8576-774295798ccc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b5f5f78825d5490bbe170255f4c85070', 'request_id': 'req-d21b2d02-a74d-403b-8576-774295798ccc'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:25.311360 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fbcad1ca-78b5-43b8-bfa0-3d6144ae6b52 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ceddaa716eb94e2cbcca0eaa1d6096fc'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:25.380816 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d21b2d02-a74d-403b-8576-774295798ccc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b5f5f78825d5490bbe170255f4c85070', 'request_id': 'req-d21b2d02-a74d-403b-8576-774295798ccc'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:25.387458 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d21b2d02-a74d-403b-8576-774295798ccc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b5f5f78825d5490bbe170255f4c85070', 'request_id': 'req-d21b2d02-a74d-403b-8576-774295798ccc'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:25.401678 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d21b2d02-a74d-403b-8576-774295798ccc admin admin] The token cache is being invalidate because project b5f5f78825d5490bbe170255f4c85070 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:25.402050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d21b2d02-a74d-403b-8576-774295798ccc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:25.402694 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d21b2d02-a74d-403b-8576-774295798ccc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:25.404235 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1074/2165] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:24 2022] DELETE /identity/v3/projects/b5f5f78825d5490bbe170255f4c85070 => generated 0 bytes in 521 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:25.404980 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21b2d02-a74d-403b-8576-774295798ccc admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:25.407523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d21b2d02-a74d-403b-8576-774295798ccc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:25.415903 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:25.499246 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fbcad1ca-78b5-43b8-bfa0-3d6144ae6b52 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ceddaa716eb94e2cbcca0eaa1d6096fc'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:25.520799 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fbcad1ca-78b5-43b8-bfa0-3d6144ae6b52 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ceddaa716eb94e2cbcca0eaa1d6096fc'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:25.521867 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fbcad1ca-78b5-43b8-bfa0-3d6144ae6b52 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ceddaa716eb94e2cbcca0eaa1d6096fc'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:25.530797 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21b2d02-a74d-403b-8576-774295798ccc admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:25.536366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fbcad1ca-78b5-43b8-bfa0-3d6144ae6b52 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ceddaa716eb94e2cbcca0eaa1d6096fc'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:25.538530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dfcee4-6b10-4775-9e17-78e957c97bc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:25.539492 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dfcee4-6b10-4775-9e17-78e957c97bc0 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:25.539835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dfcee4-6b10-4775-9e17-78e957c97bc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:25.540233 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dfcee4-6b10-4775-9e17-78e957c97bc0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:25.541068 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08dfcee4-6b10-4775-9e17-78e957c97bc0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:25.543630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08dfcee4-6b10-4775-9e17-78e957c97bc0 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:25.581110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1092/2166] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:25 2022] DELETE /identity/v3/users/ceddaa716eb94e2cbcca0eaa1d6096fc => generated 0 bytes in 569 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:25.581110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcad1ca-78b5-43b8-bfa0-3d6144ae6b52 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:25.585924 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-fbcad1ca-78b5-43b8-bfa0-3d6144ae6b52 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:25.648620 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1075/2167] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:25 2022] POST /identity/v3/projects => generated 379 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:25.650567 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dfcee4-6b10-4775-9e17-78e957c97bc0 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:25.656633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-08dfcee4-6b10-4775-9e17-78e957c97bc0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:25.705375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcad1ca-78b5-43b8-bfa0-3d6144ae6b52 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:25.772539 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dfcee4-6b10-4775-9e17-78e957c97bc0 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:25.781416 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad870a3c-f9db-42d8-8432-4984d0213cb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:25.782071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad870a3c-f9db-42d8-8432-4984d0213cb7 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:25.782368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad870a3c-f9db-42d8-8432-4984d0213cb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:25.782643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad870a3c-f9db-42d8-8432-4984d0213cb7 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:25.783305 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad870a3c-f9db-42d8-8432-4984d0213cb7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:25.786567 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad870a3c-f9db-42d8-8432-4984d0213cb7 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:25.810812 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389586ab-632e-4396-9751-d1a04cc8a3c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:25.811573 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389586ab-632e-4396-9751-d1a04cc8a3c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:25.812045 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389586ab-632e-4396-9751-d1a04cc8a3c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:25.814006 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389586ab-632e-4396-9751-d1a04cc8a3c7 admin admin] PATH_INFO: `/v3/users/e4673fba9c9049b5b6106e70decb1082` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:25.814529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389586ab-632e-4396-9751-d1a04cc8a3c7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4673fba9c9049b5b6106e70decb1082)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:25.816613 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389586ab-632e-4396-9751-d1a04cc8a3c7 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:25.889430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1076/2168] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:25 2022] POST /identity/v3/users => generated 296 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:25.894021 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad870a3c-f9db-42d8-8432-4984d0213cb7 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:25.904877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad870a3c-f9db-42d8-8432-4984d0213cb7 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:25.913407 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7988a72a-f8d8-4f9e-8ef3-08fe4a9a40df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:25.914016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7988a72a-f8d8-4f9e-8ef3-08fe4a9a40df admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:25.914363 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7988a72a-f8d8-4f9e-8ef3-08fe4a9a40df admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:25.914644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7988a72a-f8d8-4f9e-8ef3-08fe4a9a40df admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:25.915296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7988a72a-f8d8-4f9e-8ef3-08fe4a9a40df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:25.917024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7988a72a-f8d8-4f9e-8ef3-08fe4a9a40df admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:25.928842 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1077/2169] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:25 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:25.932879 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7988a72a-f8d8-4f9e-8ef3-08fe4a9a40df admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:25.945518 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7988a72a-f8d8-4f9e-8ef3-08fe4a9a40df admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:25.948766 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a3fef1-cf30-41cf-b15b-1538168df704 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:25.949496 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a3fef1-cf30-41cf-b15b-1538168df704 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:25.949843 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a3fef1-cf30-41cf-b15b-1538168df704 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:25.950247 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a3fef1-cf30-41cf-b15b-1538168df704 admin admin] PATH_INFO: `/v3/projects/f2789f9cb0bd44aaa264d2e76944cbc9/users/de28bb95bc464874a7f70b33e5c5f8e3/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:25.966188 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a3fef1-cf30-41cf-b15b-1538168df704 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2789f9cb0bd44aaa264d2e76944cbc9, user_id=de28bb95bc464874a7f70b33e5c5f8e3, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:25.969967 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a3fef1-cf30-41cf-b15b-1538168df704 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:26.010817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-389586ab-632e-4396-9751-d1a04cc8a3c7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4673fba9c9049b5b6106e70decb1082'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:26.117617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-389586ab-632e-4396-9751-d1a04cc8a3c7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4673fba9c9049b5b6106e70decb1082'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:26.119130 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1078/2170] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:25 2022] PUT /identity/v3/projects/f2789f9cb0bd44aaa264d2e76944cbc9/users/de28bb95bc464874a7f70b33e5c5f8e3/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:26.123812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a3fef1-cf30-41cf-b15b-1538168df704 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:26.127154 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:26.151603 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-389586ab-632e-4396-9751-d1a04cc8a3c7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4673fba9c9049b5b6106e70decb1082'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:26.152670 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-389586ab-632e-4396-9751-d1a04cc8a3c7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4673fba9c9049b5b6106e70decb1082'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:26.179563 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-389586ab-632e-4396-9751-d1a04cc8a3c7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4673fba9c9049b5b6106e70decb1082'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:26.191861 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1093/2171] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:25 2022] DELETE /identity/v3/users/e4673fba9c9049b5b6106e70decb1082 => generated 0 bytes in 614 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:26.255514 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a3fef1-cf30-41cf-b15b-1538168df704 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:26.307662 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389586ab-632e-4396-9751-d1a04cc8a3c7 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:26.309502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-389586ab-632e-4396-9751-d1a04cc8a3c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:26.327868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:DELETE=12,keystone:SELECT=85,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:26.371338 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116a23e9-8bf2-4f91-99d2-d2d8793a3dce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:26.373172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116a23e9-8bf2-4f91-99d2-d2d8793a3dce admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:26.373836 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116a23e9-8bf2-4f91-99d2-d2d8793a3dce admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:26.374104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116a23e9-8bf2-4f91-99d2-d2d8793a3dce admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:26.375804 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116a23e9-8bf2-4f91-99d2-d2d8793a3dce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:26.378910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116a23e9-8bf2-4f91-99d2-d2d8793a3dce admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:26.393930 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1079/2172] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:26 2022] GET /identity/v3/roles => generated 1420 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:03:26.395606 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116a23e9-8bf2-4f91-99d2-d2d8793a3dce admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:26.397545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-116a23e9-8bf2-4f91-99d2-d2d8793a3dce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:26.415133 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116a23e9-8bf2-4f91-99d2-d2d8793a3dce admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:26.420349 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a91a7c-c16f-4cd3-9039-ffea64186fdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:26.421069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a91a7c-c16f-4cd3-9039-ffea64186fdc admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:26.421398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a91a7c-c16f-4cd3-9039-ffea64186fdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:26.422208 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a91a7c-c16f-4cd3-9039-ffea64186fdc admin admin] PATH_INFO: `/v3/projects/f2789f9cb0bd44aaa264d2e76944cbc9/users/de28bb95bc464874a7f70b33e5c5f8e3/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:26.424503 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a91a7c-c16f-4cd3-9039-ffea64186fdc admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2789f9cb0bd44aaa264d2e76944cbc9, user_id=de28bb95bc464874a7f70b33e5c5f8e3, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:26.427231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a91a7c-c16f-4cd3-9039-ffea64186fdc admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:26.436327 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389586ab-632e-4396-9751-d1a04cc8a3c7 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:26.442373 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1080/2173] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:26 2022] PUT /identity/v3/projects/f2789f9cb0bd44aaa264d2e76944cbc9/users/de28bb95bc464874a7f70b33e5c5f8e3/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:26.545663 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c63a37-4117-4cda-9f42-f5ae9ea74535 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:26.548834 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c63a37-4117-4cda-9f42-f5ae9ea74535 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:26.549296 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c63a37-4117-4cda-9f42-f5ae9ea74535 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:26.549674 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c63a37-4117-4cda-9f42-f5ae9ea74535 admin admin] PATH_INFO: `/v3/projects/0663e86bf6e048d4bee2938e9ee39906` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:26.551042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c63a37-4117-4cda-9f42-f5ae9ea74535 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0663e86bf6e048d4bee2938e9ee39906)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:26.560382 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c63a37-4117-4cda-9f42-f5ae9ea74535 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:26.728356 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a91a7c-c16f-4cd3-9039-ffea64186fdc admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:26.730606 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d9a91a7c-c16f-4cd3-9039-ffea64186fdc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:26.763206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-39c63a37-4117-4cda-9f42-f5ae9ea74535 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0663e86bf6e048d4bee2938e9ee39906', 'request_id': 'req-39c63a37-4117-4cda-9f42-f5ae9ea74535'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:26.767145 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a91a7c-c16f-4cd3-9039-ffea64186fdc admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:26.838351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-39c63a37-4117-4cda-9f42-f5ae9ea74535 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0663e86bf6e048d4bee2938e9ee39906', 'request_id': 'req-39c63a37-4117-4cda-9f42-f5ae9ea74535'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:26.850875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-39c63a37-4117-4cda-9f42-f5ae9ea74535 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0663e86bf6e048d4bee2938e9ee39906', 'request_id': 'req-39c63a37-4117-4cda-9f42-f5ae9ea74535'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:26.858883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3218836-b4bb-419d-bdfe-ef1f61305f48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:26.860491 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3218836-b4bb-419d-bdfe-ef1f61305f48 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:26.861055 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3218836-b4bb-419d-bdfe-ef1f61305f48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:26.862143 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3218836-b4bb-419d-bdfe-ef1f61305f48 admin admin] PATH_INFO: `/v3/projects/a4c2dcf8f82b4dfaa46dca9d6851bac1` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:26.864083 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3218836-b4bb-419d-bdfe-ef1f61305f48 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a4c2dcf8f82b4dfaa46dca9d6851bac1)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:26.869602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3218836-b4bb-419d-bdfe-ef1f61305f48 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:26.869838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-39c63a37-4117-4cda-9f42-f5ae9ea74535 admin admin] The token cache is being invalidate because project 0663e86bf6e048d4bee2938e9ee39906 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:26.869838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-39c63a37-4117-4cda-9f42-f5ae9ea74535 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:26.869838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-39c63a37-4117-4cda-9f42-f5ae9ea74535 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:26.870355 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1094/2174] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:26 2022] DELETE /identity/v3/projects/0663e86bf6e048d4bee2938e9ee39906 => generated 0 bytes in 563 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:26.875109 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e74cb5d-fda1-47e1-824d-432551fb4f06 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:26.875478 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e74cb5d-fda1-47e1-824d-432551fb4f06 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:26.876461 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e74cb5d-fda1-47e1-824d-432551fb4f06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:26.910795 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-1e74cb5d-fda1-47e1-824d-432551fb4f06 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:26.912026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-1e74cb5d-fda1-47e1-824d-432551fb4f06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:26.941685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1095/2175] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:26.943575 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91136cfd-f4a9-4fb4-924f-fc99448bd71e None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:26.944374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91136cfd-f4a9-4fb4-924f-fc99448bd71e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:26.944721 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91136cfd-f4a9-4fb4-924f-fc99448bd71e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:26.989011 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-91136cfd-f4a9-4fb4-924f-fc99448bd71e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:26.990369 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-91136cfd-f4a9-4fb4-924f-fc99448bd71e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:27.001015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f3218836-b4bb-419d-bdfe-ef1f61305f48 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a4c2dcf8f82b4dfaa46dca9d6851bac1', 'request_id': 'req-f3218836-b4bb-419d-bdfe-ef1f61305f48'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:27.069474 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1096/2176] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:27.073608 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91136cfd-f4a9-4fb4-924f-fc99448bd71e None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:27.077940 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f3218836-b4bb-419d-bdfe-ef1f61305f48 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a4c2dcf8f82b4dfaa46dca9d6851bac1', 'request_id': 'req-f3218836-b4bb-419d-bdfe-ef1f61305f48'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:27.088229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91136cfd-f4a9-4fb4-924f-fc99448bd71e None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:27.088586 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f3218836-b4bb-419d-bdfe-ef1f61305f48 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a4c2dcf8f82b4dfaa46dca9d6851bac1', 'request_id': 'req-f3218836-b4bb-419d-bdfe-ef1f61305f48'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:27.089489 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928255e6-132d-471a-a4d2-bcf2ee88da71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:27.090105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928255e6-132d-471a-a4d2-bcf2ee88da71 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:27.090486 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928255e6-132d-471a-a4d2-bcf2ee88da71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:27.090770 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-928255e6-132d-471a-a4d2-bcf2ee88da71 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:27.091438 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928255e6-132d-471a-a4d2-bcf2ee88da71 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:03:27.092052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928255e6-132d-471a-a4d2-bcf2ee88da71 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:27.093990 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-928255e6-132d-471a-a4d2-bcf2ee88da71 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:27.103704 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f3218836-b4bb-419d-bdfe-ef1f61305f48 admin admin] The token cache is being invalidate because project a4c2dcf8f82b4dfaa46dca9d6851bac1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:27.103704 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f3218836-b4bb-419d-bdfe-ef1f61305f48 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:27.104505 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f3218836-b4bb-419d-bdfe-ef1f61305f48 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:27.106523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1081/2177] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:26 2022] DELETE /identity/v3/projects/a4c2dcf8f82b4dfaa46dca9d6851bac1 => generated 0 bytes in 379 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:27.112534 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0797f5-f487-4d03-922f-6eb68cd7ee7e None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:27.112883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0797f5-f487-4d03-922f-6eb68cd7ee7e None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:27.113170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0797f5-f487-4d03-922f-6eb68cd7ee7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:27.113271 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1097/2178] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:03:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:03:27.120086 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928255e6-132d-471a-a4d2-bcf2ee88da71 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:27.123346 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-928255e6-132d-471a-a4d2-bcf2ee88da71 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:27.135747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-9f0797f5-f487-4d03-922f-6eb68cd7ee7e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:27.136890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9f0797f5-f487-4d03-922f-6eb68cd7ee7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:27.143208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-928255e6-132d-471a-a4d2-bcf2ee88da71 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:27.147386 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1082/2179] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:27.154141 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69773699-f26a-45c1-a31d-3685e201052b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:27.154141 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69773699-f26a-45c1-a31d-3685e201052b admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:27.154141 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69773699-f26a-45c1-a31d-3685e201052b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:27.154141 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69773699-f26a-45c1-a31d-3685e201052b admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:27.154522 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69773699-f26a-45c1-a31d-3685e201052b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:27.154522 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69773699-f26a-45c1-a31d-3685e201052b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:27.156288 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c04ffb-1519-49d5-a598-4233285ee01f None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:27.157826 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c04ffb-1519-49d5-a598-4233285ee01f None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:27.158108 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c04ffb-1519-49d5-a598-4233285ee01f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:27.191880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-38c04ffb-1519-49d5-a598-4233285ee01f None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:27.193113 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-38c04ffb-1519-49d5-a598-4233285ee01f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:27.205244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1083/2180] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:27.209831 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c04ffb-1519-49d5-a598-4233285ee01f None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:27.221817 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c04ffb-1519-49d5-a598-4233285ee01f None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:27.227788 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a11b47-dc7e-4a00-b4c3-264431bd9fc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:27.228581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a11b47-dc7e-4a00-b4c3-264431bd9fc4 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:27.229006 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a11b47-dc7e-4a00-b4c3-264431bd9fc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:27.229287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a11b47-dc7e-4a00-b4c3-264431bd9fc4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:27.229946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a11b47-dc7e-4a00-b4c3-264431bd9fc4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:03:27.230396 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a11b47-dc7e-4a00-b4c3-264431bd9fc4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:27.231868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a11b47-dc7e-4a00-b4c3-264431bd9fc4 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:27.249455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:27.249455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1084/2181] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:03:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 14:03:27.251633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a11b47-dc7e-4a00-b4c3-264431bd9fc4 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:27.261889 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a11b47-dc7e-4a00-b4c3-264431bd9fc4 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:27.269352 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7280309e-2e88-4f21-9f99-bd7a41a4cf37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:27.270393 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7280309e-2e88-4f21-9f99-bd7a41a4cf37 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:27.270393 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7280309e-2e88-4f21-9f99-bd7a41a4cf37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:27.277840 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7280309e-2e88-4f21-9f99-bd7a41a4cf37 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:27.277840 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7280309e-2e88-4f21-9f99-bd7a41a4cf37 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:27.277840 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7280309e-2e88-4f21-9f99-bd7a41a4cf37 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:27.288401 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1098/2182] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:27 2022] POST /identity/v3/projects => generated 375 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:27.289456 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69773699-f26a-45c1-a31d-3685e201052b admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:27.366378 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69773699-f26a-45c1-a31d-3685e201052b admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:27.371263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d89565d-952b-40ce-ae01-09fe78793170 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:27.371923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d89565d-952b-40ce-ae01-09fe78793170 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:27.379093 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d89565d-952b-40ce-ae01-09fe78793170 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:27.379093 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d89565d-952b-40ce-ae01-09fe78793170 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:27.379093 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d89565d-952b-40ce-ae01-09fe78793170 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:27.379093 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d89565d-952b-40ce-ae01-09fe78793170 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:27.477291 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1085/2183] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:27 2022] POST /identity/v3/projects => generated 379 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:03:27.478623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7280309e-2e88-4f21-9f99-bd7a41a4cf37 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:27.492802 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1099/2184] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:27 2022] POST /identity/v3/users => generated 288 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:27.502639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d89565d-952b-40ce-ae01-09fe78793170 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:27.518453 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d89565d-952b-40ce-ae01-09fe78793170 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:27.648417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7280309e-2e88-4f21-9f99-bd7a41a4cf37 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:27.659632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e606c89c-a0b9-4ae9-8760-56dd616150bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:27.660449 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e606c89c-a0b9-4ae9-8760-56dd616150bd admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:27.661983 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e606c89c-a0b9-4ae9-8760-56dd616150bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:27.662292 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e606c89c-a0b9-4ae9-8760-56dd616150bd admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:27.662951 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e606c89c-a0b9-4ae9-8760-56dd616150bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:27.665675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e606c89c-a0b9-4ae9-8760-56dd616150bd admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:27.673169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb5f175-0247-4bf2-8b04-f634af6a9c88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:27.673676 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb5f175-0247-4bf2-8b04-f634af6a9c88 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:27.674629 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb5f175-0247-4bf2-8b04-f634af6a9c88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:27.674928 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb5f175-0247-4bf2-8b04-f634af6a9c88 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:27.675741 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb5f175-0247-4bf2-8b04-f634af6a9c88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:27.681055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb5f175-0247-4bf2-8b04-f634af6a9c88 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:27.701366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1100/2185] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:27 2022] GET /identity/v3/roles => generated 1420 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:27.702307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb5f175-0247-4bf2-8b04-f634af6a9c88 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:27.723644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb5f175-0247-4bf2-8b04-f634af6a9c88 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:27.729516 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cffb083-c5d7-4e27-ab7b-d4220ed021dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:27.730644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cffb083-c5d7-4e27-ab7b-d4220ed021dc admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:27.730644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cffb083-c5d7-4e27-ab7b-d4220ed021dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:27.730953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cffb083-c5d7-4e27-ab7b-d4220ed021dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:27.731407 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cffb083-c5d7-4e27-ab7b-d4220ed021dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:27.733371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cffb083-c5d7-4e27-ab7b-d4220ed021dc admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:27.757749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1101/2186] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:27 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:27.758736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cffb083-c5d7-4e27-ab7b-d4220ed021dc admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:27.772763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cffb083-c5d7-4e27-ab7b-d4220ed021dc admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:27.774012 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1086/2187] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:27 2022] POST /identity/v3/users => generated 290 bytes in 292 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:27.775474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e606c89c-a0b9-4ae9-8760-56dd616150bd admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:27.780328 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032ba502-1f2f-4f7a-acd7-2462d12ca28b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:27.781174 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032ba502-1f2f-4f7a-acd7-2462d12ca28b admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:27.782395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032ba502-1f2f-4f7a-acd7-2462d12ca28b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:27.782690 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032ba502-1f2f-4f7a-acd7-2462d12ca28b admin admin] PATH_INFO: `/v3/projects/da2bf84b8d644402b31dfe09fab21dab/users/f0b27af5fff04c8386c3578939887a4d/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:27.786727 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e606c89c-a0b9-4ae9-8760-56dd616150bd admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:27.791906 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ef99aa-1c40-4008-a204-d6d2e25dd72f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:27.792791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ef99aa-1c40-4008-a204-d6d2e25dd72f admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:27.793072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ef99aa-1c40-4008-a204-d6d2e25dd72f admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:27.793353 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ef99aa-1c40-4008-a204-d6d2e25dd72f admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:27.804533 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032ba502-1f2f-4f7a-acd7-2462d12ca28b admin admin] RBAC: Authorizing `identity:create_grant(project_id=da2bf84b8d644402b31dfe09fab21dab, user_id=f0b27af5fff04c8386c3578939887a4d, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:27.807421 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032ba502-1f2f-4f7a-acd7-2462d12ca28b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:27.808378 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ef99aa-1c40-4008-a204-d6d2e25dd72f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:27.810448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ef99aa-1c40-4008-a204-d6d2e25dd72f admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:27.829871 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1087/2188] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:27 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:27.831012 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ef99aa-1c40-4008-a204-d6d2e25dd72f admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:27.847586 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ef99aa-1c40-4008-a204-d6d2e25dd72f admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:27.859524 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fca625-f76e-4197-b6ed-69c589c37cea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:27.859524 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fca625-f76e-4197-b6ed-69c589c37cea admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:27.863694 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fca625-f76e-4197-b6ed-69c589c37cea admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:27.864241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fca625-f76e-4197-b6ed-69c589c37cea admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:27.869876 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fca625-f76e-4197-b6ed-69c589c37cea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:27.878526 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fca625-f76e-4197-b6ed-69c589c37cea admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:27.885572 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1102/2189] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:27 2022] PUT /identity/v3/projects/da2bf84b8d644402b31dfe09fab21dab/users/f0b27af5fff04c8386c3578939887a4d/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:27.894795 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1088/2190] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:27 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:27.899068 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032ba502-1f2f-4f7a-acd7-2462d12ca28b admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:27.908041 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e011977-2cc9-49ba-a712-08f2529c8028 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:27.913798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e011977-2cc9-49ba-a712-08f2529c8028 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:27.913798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e011977-2cc9-49ba-a712-08f2529c8028 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:27.990154 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032ba502-1f2f-4f7a-acd7-2462d12ca28b admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:27.998868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-8e011977-2cc9-49ba-a712-08f2529c8028 None None] MFA Rules not processed for user `f0b27af5fff04c8386c3578939887a4d`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:28.009941 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8e011977-2cc9-49ba-a712-08f2529c8028 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:28.011550 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27043d4e-b0a8-4d7e-a390-af5c14dcee31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:28.012236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27043d4e-b0a8-4d7e-a390-af5c14dcee31 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:28.012515 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27043d4e-b0a8-4d7e-a390-af5c14dcee31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:28.012786 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27043d4e-b0a8-4d7e-a390-af5c14dcee31 admin admin] PATH_INFO: `/v3/projects/9ea9d1031f974e8a8919d09372f5c3f8/users/10983593b3af4aabac4a08d189a8e3eb/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:28.074930 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27043d4e-b0a8-4d7e-a390-af5c14dcee31 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9ea9d1031f974e8a8919d09372f5c3f8, user_id=10983593b3af4aabac4a08d189a8e3eb, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:28.083255 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27043d4e-b0a8-4d7e-a390-af5c14dcee31 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:28.095825 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:28.152313 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1089/2191] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:27 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:28.153636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e011977-2cc9-49ba-a712-08f2529c8028 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:28.155164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8e011977-2cc9-49ba-a712-08f2529c8028 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:28.259878 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1103/2192] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:27 2022] PUT /identity/v3/projects/9ea9d1031f974e8a8919d09372f5c3f8/users/10983593b3af4aabac4a08d189a8e3eb/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:28.262274 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7451dfd-d2ee-40d7-8f00-e5d048363c34 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:28.262855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7451dfd-d2ee-40d7-8f00-e5d048363c34 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:28.263278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7451dfd-d2ee-40d7-8f00-e5d048363c34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:28.280579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e011977-2cc9-49ba-a712-08f2529c8028 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:28.350527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-a7451dfd-d2ee-40d7-8f00-e5d048363c34 None None] MFA Rules not processed for user `10983593b3af4aabac4a08d189a8e3eb`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:28.351832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a7451dfd-d2ee-40d7-8f00-e5d048363c34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:28.432199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74704b8-abe8-41e8-84ce-0ff7767df021 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:28.432951 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74704b8-abe8-41e8-84ce-0ff7767df021 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:28.433228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74704b8-abe8-41e8-84ce-0ff7767df021 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:28.433541 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74704b8-abe8-41e8-84ce-0ff7767df021 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:28.487721 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1104/2193] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:28 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:28.492308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7451dfd-d2ee-40d7-8f00-e5d048363c34 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:28.580759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74704b8-abe8-41e8-84ce-0ff7767df021 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:28.583120 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74704b8-abe8-41e8-84ce-0ff7767df021 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:28.595082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1090/2194] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:28 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:28.629376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7451dfd-d2ee-40d7-8f00-e5d048363c34 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:28.648198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad1af3d-74de-4bd3-9d1e-b6659e9d0264 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:28.648198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad1af3d-74de-4bd3-9d1e-b6659e9d0264 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:28.648198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad1af3d-74de-4bd3-9d1e-b6659e9d0264 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:28.648198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad1af3d-74de-4bd3-9d1e-b6659e9d0264 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:28.648544 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad1af3d-74de-4bd3-9d1e-b6659e9d0264 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:28.649865 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad1af3d-74de-4bd3-9d1e-b6659e9d0264 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:28.768036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1105/2195] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:28 2022] POST /identity/v3/projects => generated 381 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:03:28.771714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74704b8-abe8-41e8-84ce-0ff7767df021 service cinder] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:28.795631 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74704b8-abe8-41e8-84ce-0ff7767df021 service cinder] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:28.912402 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3310b2ae-9e18-440f-a615-03f502a95223 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:28.918151 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3310b2ae-9e18-440f-a615-03f502a95223 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:28.919205 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3310b2ae-9e18-440f-a615-03f502a95223 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:28.919730 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3310b2ae-9e18-440f-a615-03f502a95223 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:28.921066 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3310b2ae-9e18-440f-a615-03f502a95223 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:28.924567 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3310b2ae-9e18-440f-a615-03f502a95223 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:29.043704 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:29.087725 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1091/2196] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:28 2022] POST /identity/v3/users => generated 297 bytes in 317 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:29.088928 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad1af3d-74de-4bd3-9d1e-b6659e9d0264 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:29.105291 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad1af3d-74de-4bd3-9d1e-b6659e9d0264 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:29.111097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862de3f6-7e7c-4e36-9afd-47d95a82582c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:29.111960 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862de3f6-7e7c-4e36-9afd-47d95a82582c admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:29.112286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862de3f6-7e7c-4e36-9afd-47d95a82582c admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:29.112558 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-862de3f6-7e7c-4e36-9afd-47d95a82582c admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:29.113210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862de3f6-7e7c-4e36-9afd-47d95a82582c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:29.115027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-862de3f6-7e7c-4e36-9afd-47d95a82582c admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:29.129029 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1106/2197] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:29 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:03:29.133405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3310b2ae-9e18-440f-a615-03f502a95223 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:29.145603 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3310b2ae-9e18-440f-a615-03f502a95223 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:29.150666 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1549b809-3385-4081-981c-b6e7b605fdc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:29.151564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1549b809-3385-4081-981c-b6e7b605fdc7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:29.151948 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1549b809-3385-4081-981c-b6e7b605fdc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:29.152384 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1549b809-3385-4081-981c-b6e7b605fdc7 admin admin] PATH_INFO: `/v3/projects/ea6bd75825f949e9ba9236ac34ceda4d/users/a9a3585b7fb3445bab17eb682b3d32b0/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:29.186242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1549b809-3385-4081-981c-b6e7b605fdc7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea6bd75825f949e9ba9236ac34ceda4d, user_id=a9a3585b7fb3445bab17eb682b3d32b0, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:29.189728 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1549b809-3385-4081-981c-b6e7b605fdc7 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:29.277493 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1092/2198] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:29 2022] PUT /identity/v3/projects/ea6bd75825f949e9ba9236ac34ceda4d/users/a9a3585b7fb3445bab17eb682b3d32b0/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:29.283423 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862de3f6-7e7c-4e36-9afd-47d95a82582c admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:29.295623 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:29.311019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-862de3f6-7e7c-4e36-9afd-47d95a82582c admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:29.412614 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e7aeaa-754e-4a4b-9087-13b141505ea5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:29.413309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e7aeaa-754e-4a4b-9087-13b141505ea5 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:29.413658 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e7aeaa-754e-4a4b-9087-13b141505ea5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:29.414005 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e7aeaa-754e-4a4b-9087-13b141505ea5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:29.415548 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e7aeaa-754e-4a4b-9087-13b141505ea5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:29.417349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e7aeaa-754e-4a4b-9087-13b141505ea5 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:29.438824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1107/2199] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:29 2022] GET /identity/v3/roles => generated 1420 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:29.443062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1549b809-3385-4081-981c-b6e7b605fdc7 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:29.471427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1549b809-3385-4081-981c-b6e7b605fdc7 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:29.481863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f44d3ef-2905-4a33-be3f-f5f1e02c2ba9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:29.482692 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f44d3ef-2905-4a33-be3f-f5f1e02c2ba9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:29.483074 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f44d3ef-2905-4a33-be3f-f5f1e02c2ba9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:29.483451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f44d3ef-2905-4a33-be3f-f5f1e02c2ba9 admin admin] PATH_INFO: `/v3/projects/ea6bd75825f949e9ba9236ac34ceda4d/users/a9a3585b7fb3445bab17eb682b3d32b0/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:29.489416 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f44d3ef-2905-4a33-be3f-f5f1e02c2ba9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea6bd75825f949e9ba9236ac34ceda4d, user_id=a9a3585b7fb3445bab17eb682b3d32b0, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:29.492142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f44d3ef-2905-4a33-be3f-f5f1e02c2ba9 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:29.512997 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1093/2200] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:29 2022] PUT /identity/v3/projects/ea6bd75825f949e9ba9236ac34ceda4d/users/a9a3585b7fb3445bab17eb682b3d32b0/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:29.525056 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb72720e-2a9e-4085-9dc2-2d443f48f9d2 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:29.525345 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb72720e-2a9e-4085-9dc2-2d443f48f9d2 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:29.525621 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb72720e-2a9e-4085-9dc2-2d443f48f9d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:29.634038 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-eb72720e-2a9e-4085-9dc2-2d443f48f9d2 None None] MFA Rules not processed for user `a9a3585b7fb3445bab17eb682b3d32b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:29.635590 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-eb72720e-2a9e-4085-9dc2-2d443f48f9d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:29.851654 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1108/2201] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:29 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 328 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:29.864425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f44d3ef-2905-4a33-be3f-f5f1e02c2ba9 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:30.058456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f44d3ef-2905-4a33-be3f-f5f1e02c2ba9 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:30.067258 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35720bab-3a8c-411b-9668-31c34b78be52 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:30.068093 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35720bab-3a8c-411b-9668-31c34b78be52 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:30.068518 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35720bab-3a8c-411b-9668-31c34b78be52 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:30.068933 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35720bab-3a8c-411b-9668-31c34b78be52 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:30.086250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35720bab-3a8c-411b-9668-31c34b78be52 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:30.089962 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35720bab-3a8c-411b-9668-31c34b78be52 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:30.106947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1094/2202] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:29 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 243 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:30.237654 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:30.726674 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb72720e-2a9e-4085-9dc2-2d443f48f9d2 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:30.882567 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb72720e-2a9e-4085-9dc2-2d443f48f9d2 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:30.888547 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec44bb43-bfcd-44cb-9e0b-ccefe92b4ef2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:30.889520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec44bb43-bfcd-44cb-9e0b-ccefe92b4ef2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:30.890893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec44bb43-bfcd-44cb-9e0b-ccefe92b4ef2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:30.896714 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec44bb43-bfcd-44cb-9e0b-ccefe92b4ef2 admin admin] PATH_INFO: `/v3/users/10983593b3af4aabac4a08d189a8e3eb` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:30.896714 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec44bb43-bfcd-44cb-9e0b-ccefe92b4ef2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=10983593b3af4aabac4a08d189a8e3eb)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:30.896714 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec44bb43-bfcd-44cb-9e0b-ccefe92b4ef2 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:31.040689 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:31.153195 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ec44bb43-bfcd-44cb-9e0b-ccefe92b4ef2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '10983593b3af4aabac4a08d189a8e3eb'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:31.275383 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ec44bb43-bfcd-44cb-9e0b-ccefe92b4ef2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '10983593b3af4aabac4a08d189a8e3eb'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:31.301430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ec44bb43-bfcd-44cb-9e0b-ccefe92b4ef2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '10983593b3af4aabac4a08d189a8e3eb'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:31.304618 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ec44bb43-bfcd-44cb-9e0b-ccefe92b4ef2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '10983593b3af4aabac4a08d189a8e3eb'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:31.327153 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ec44bb43-bfcd-44cb-9e0b-ccefe92b4ef2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '10983593b3af4aabac4a08d189a8e3eb'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:31.344753 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1109/2203] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:30 2022] DELETE /identity/v3/users/10983593b3af4aabac4a08d189a8e3eb => generated 0 bytes in 614 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:31.344941 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a12a235-a930-40ed-80ef-8dde2cba78de None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:31.345399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a12a235-a930-40ed-80ef-8dde2cba78de None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:31.345399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a12a235-a930-40ed-80ef-8dde2cba78de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:31.391698 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-0a12a235-a930-40ed-80ef-8dde2cba78de None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:31.393112 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-0a12a235-a930-40ed-80ef-8dde2cba78de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:31.510436 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1095/2204] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:31.516543 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec44bb43-bfcd-44cb-9e0b-ccefe92b4ef2 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:31.518204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-ec44bb43-bfcd-44cb-9e0b-ccefe92b4ef2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:31.628743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec44bb43-bfcd-44cb-9e0b-ccefe92b4ef2 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:31.656864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3b07e4-bc96-4a10-bdb7-fe933f462620 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:31.657455 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3b07e4-bc96-4a10-bdb7-fe933f462620 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:31.658058 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3b07e4-bc96-4a10-bdb7-fe933f462620 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:31.658058 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3b07e4-bc96-4a10-bdb7-fe933f462620 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:31.679167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3b07e4-bc96-4a10-bdb7-fe933f462620 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:31.689405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3b07e4-bc96-4a10-bdb7-fe933f462620 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:31.704022 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1110/2205] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:31.718603 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:32.529276 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:33.090744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a12a235-a930-40ed-80ef-8dde2cba78de None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:33.092670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-0a12a235-a930-40ed-80ef-8dde2cba78de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:33.133807 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a12a235-a930-40ed-80ef-8dde2cba78de None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:33.144586 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5713c9b-9dfc-4a91-a4dc-f15738db8a04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:33.145795 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5713c9b-9dfc-4a91-a4dc-f15738db8a04 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:33.146295 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5713c9b-9dfc-4a91-a4dc-f15738db8a04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:33.147065 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5713c9b-9dfc-4a91-a4dc-f15738db8a04 admin admin] PATH_INFO: `/v3/projects/9ea9d1031f974e8a8919d09372f5c3f8` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:33.148660 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5713c9b-9dfc-4a91-a4dc-f15738db8a04 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9ea9d1031f974e8a8919d09372f5c3f8)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:33.152022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5713c9b-9dfc-4a91-a4dc-f15738db8a04 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:33.435758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d5713c9b-9dfc-4a91-a4dc-f15738db8a04 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9ea9d1031f974e8a8919d09372f5c3f8', 'request_id': 'req-d5713c9b-9dfc-4a91-a4dc-f15738db8a04'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:33.545957 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d5713c9b-9dfc-4a91-a4dc-f15738db8a04 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9ea9d1031f974e8a8919d09372f5c3f8', 'request_id': 'req-d5713c9b-9dfc-4a91-a4dc-f15738db8a04'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:33.554416 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d5713c9b-9dfc-4a91-a4dc-f15738db8a04 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9ea9d1031f974e8a8919d09372f5c3f8', 'request_id': 'req-d5713c9b-9dfc-4a91-a4dc-f15738db8a04'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:33.577817 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d5713c9b-9dfc-4a91-a4dc-f15738db8a04 admin admin] The token cache is being invalidate because project 9ea9d1031f974e8a8919d09372f5c3f8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:33.578116 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d5713c9b-9dfc-4a91-a4dc-f15738db8a04 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:33.584784 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d5713c9b-9dfc-4a91-a4dc-f15738db8a04 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:33.594560 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1096/2206] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:33 2022] DELETE /identity/v3/projects/9ea9d1031f974e8a8919d09372f5c3f8 => generated 0 bytes in 500 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:33.596032 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3b07e4-bc96-4a10-bdb7-fe933f462620 service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:33.620121 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3b07e4-bc96-4a10-bdb7-fe933f462620 service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:33.620828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5713c9b-9dfc-4a91-a4dc-f15738db8a04 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:33.622355 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d5713c9b-9dfc-4a91-a4dc-f15738db8a04 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:33.747720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f4de6c-db1a-4fdf-b731-8a0520a7515e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:33.748755 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f4de6c-db1a-4fdf-b731-8a0520a7515e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:33.750199 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f4de6c-db1a-4fdf-b731-8a0520a7515e admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:33.750743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f4de6c-db1a-4fdf-b731-8a0520a7515e admin admin] PATH_INFO: `/v3/users/a9a3585b7fb3445bab17eb682b3d32b0` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:33.752226 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f4de6c-db1a-4fdf-b731-8a0520a7515e admin admin] RBAC: Authorizing `identity:delete_user(user_id=a9a3585b7fb3445bab17eb682b3d32b0)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:33.756628 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f4de6c-db1a-4fdf-b731-8a0520a7515e admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:33.793458 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5713c9b-9dfc-4a91-a4dc-f15738db8a04 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:33.807640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1e56cf-ca83-4166-acd4-68c185931965 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:33.808792 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1e56cf-ca83-4166-acd4-68c185931965 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:33.809178 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1e56cf-ca83-4166-acd4-68c185931965 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:33.810707 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1e56cf-ca83-4166-acd4-68c185931965 admin admin] PATH_INFO: `/v3/users/f0b27af5fff04c8386c3578939887a4d` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:33.814995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:33.816356 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1e56cf-ca83-4166-acd4-68c185931965 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f0b27af5fff04c8386c3578939887a4d)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:33.819070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1e56cf-ca83-4166-acd4-68c185931965 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:33.892539 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:34.016156 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-61f4de6c-db1a-4fdf-b731-8a0520a7515e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a9a3585b7fb3445bab17eb682b3d32b0'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:34.051814 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2d1e56cf-ca83-4166-acd4-68c185931965 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f0b27af5fff04c8386c3578939887a4d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:34.124881 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-61f4de6c-db1a-4fdf-b731-8a0520a7515e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a9a3585b7fb3445bab17eb682b3d32b0'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:34.144401 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-61f4de6c-db1a-4fdf-b731-8a0520a7515e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a9a3585b7fb3445bab17eb682b3d32b0'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:34.145431 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-61f4de6c-db1a-4fdf-b731-8a0520a7515e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a9a3585b7fb3445bab17eb682b3d32b0'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:34.153091 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-61f4de6c-db1a-4fdf-b731-8a0520a7515e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a9a3585b7fb3445bab17eb682b3d32b0'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:34.161552 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1111/2207] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:33 2022] DELETE /identity/v3/users/a9a3585b7fb3445bab17eb682b3d32b0 => generated 0 bytes in 566 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:34.254343 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2d1e56cf-ca83-4166-acd4-68c185931965 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f0b27af5fff04c8386c3578939887a4d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:34.255854 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2d1e56cf-ca83-4166-acd4-68c185931965 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f0b27af5fff04c8386c3578939887a4d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:34.290288 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2d1e56cf-ca83-4166-acd4-68c185931965 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f0b27af5fff04c8386c3578939887a4d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:34.290288 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2d1e56cf-ca83-4166-acd4-68c185931965 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f0b27af5fff04c8386c3578939887a4d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:34.339917 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1097/2208] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:33 2022] DELETE /identity/v3/users/f0b27af5fff04c8386c3578939887a4d => generated 0 bytes in 720 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:34.345139 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b652dd3c-d2bd-4e4e-bb84-e6073c547161 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:34.345426 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b652dd3c-d2bd-4e4e-bb84-e6073c547161 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:34.345756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b652dd3c-d2bd-4e4e-bb84-e6073c547161 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:34.382580 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-b652dd3c-d2bd-4e4e-bb84-e6073c547161 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:34.383834 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-b652dd3c-d2bd-4e4e-bb84-e6073c547161 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:34.504162 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1112/2209] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:34.592495 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1e56cf-ca83-4166-acd4-68c185931965 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:34.592495 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2d1e56cf-ca83-4166-acd4-68c185931965 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:34.772008 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1e56cf-ca83-4166-acd4-68c185931965 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:34.778584 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7d9c1a-6a19-42a2-b7eb-984a9e8ec6de service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:34.779611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7d9c1a-6a19-42a2-b7eb-984a9e8ec6de service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:34.780144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7d9c1a-6a19-42a2-b7eb-984a9e8ec6de service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:34.780623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7d9c1a-6a19-42a2-b7eb-984a9e8ec6de service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:34.792778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7d9c1a-6a19-42a2-b7eb-984a9e8ec6de service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:34.794957 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7d9c1a-6a19-42a2-b7eb-984a9e8ec6de service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:34.809736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1098/2210] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:35.014688 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:35.135815 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b652dd3c-d2bd-4e4e-bb84-e6073c547161 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:35.137292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-b652dd3c-d2bd-4e4e-bb84-e6073c547161 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:35.197373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b652dd3c-d2bd-4e4e-bb84-e6073c547161 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:35.222202 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaffa916-62ec-49c4-a43e-bba7b0ec4054 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:35.226108 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaffa916-62ec-49c4-a43e-bba7b0ec4054 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:35.226108 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaffa916-62ec-49c4-a43e-bba7b0ec4054 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:35.226108 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaffa916-62ec-49c4-a43e-bba7b0ec4054 admin admin] PATH_INFO: `/v3/projects/ea6bd75825f949e9ba9236ac34ceda4d` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:35.228151 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaffa916-62ec-49c4-a43e-bba7b0ec4054 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ea6bd75825f949e9ba9236ac34ceda4d)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:35.230945 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaffa916-62ec-49c4-a43e-bba7b0ec4054 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:35.400697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-aaffa916-62ec-49c4-a43e-bba7b0ec4054 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ea6bd75825f949e9ba9236ac34ceda4d', 'request_id': 'req-aaffa916-62ec-49c4-a43e-bba7b0ec4054'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:35.494679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-aaffa916-62ec-49c4-a43e-bba7b0ec4054 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ea6bd75825f949e9ba9236ac34ceda4d', 'request_id': 'req-aaffa916-62ec-49c4-a43e-bba7b0ec4054'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:35.511193 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-aaffa916-62ec-49c4-a43e-bba7b0ec4054 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ea6bd75825f949e9ba9236ac34ceda4d', 'request_id': 'req-aaffa916-62ec-49c4-a43e-bba7b0ec4054'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:35.551243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-aaffa916-62ec-49c4-a43e-bba7b0ec4054 admin admin] The token cache is being invalidate because project ea6bd75825f949e9ba9236ac34ceda4d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:35.551539 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-aaffa916-62ec-49c4-a43e-bba7b0ec4054 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:35.557064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-aaffa916-62ec-49c4-a43e-bba7b0ec4054 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:35.571657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1113/2211] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:35 2022] DELETE /identity/v3/projects/ea6bd75825f949e9ba9236ac34ceda4d => generated 0 bytes in 436 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:35.576871 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8354cf83-cca0-4bad-b385-528edad19189 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:35.577465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8354cf83-cca0-4bad-b385-528edad19189 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:35.577560 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8354cf83-cca0-4bad-b385-528edad19189 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:35.637471 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:35.639684 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-8354cf83-cca0-4bad-b385-528edad19189 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:35.643577 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8354cf83-cca0-4bad-b385-528edad19189 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:35.834605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1099/2212] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:35.852676 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7dca2b-fb63-411f-80fd-763fe6841ef2 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:35.852676 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7dca2b-fb63-411f-80fd-763fe6841ef2 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:35.852676 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7dca2b-fb63-411f-80fd-763fe6841ef2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:35.885199 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-9f7dca2b-fb63-411f-80fd-763fe6841ef2 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:35.886382 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-9f7dca2b-fb63-411f-80fd-763fe6841ef2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:35.916966 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1114/2213] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:35.921188 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8354cf83-cca0-4bad-b385-528edad19189 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:35.940312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8354cf83-cca0-4bad-b385-528edad19189 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:35.959318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972fb1f4-1899-470a-82a3-f5d52ef721c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:35.959957 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972fb1f4-1899-470a-82a3-f5d52ef721c5 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:35.960908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972fb1f4-1899-470a-82a3-f5d52ef721c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:35.961204 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972fb1f4-1899-470a-82a3-f5d52ef721c5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:35.961884 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972fb1f4-1899-470a-82a3-f5d52ef721c5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:03:35.962361 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972fb1f4-1899-470a-82a3-f5d52ef721c5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:35.966932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972fb1f4-1899-470a-82a3-f5d52ef721c5 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:35.997110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1100/2214] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:03:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:03:36.002587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7dca2b-fb63-411f-80fd-763fe6841ef2 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:36.018211 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7dca2b-fb63-411f-80fd-763fe6841ef2 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:36.032942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:36.037689 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b090b8-9ff5-4c23-b02c-6b69d573d848 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:36.038591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b090b8-9ff5-4c23-b02c-6b69d573d848 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:36.038920 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b090b8-9ff5-4c23-b02c-6b69d573d848 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:36.039473 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b090b8-9ff5-4c23-b02c-6b69d573d848 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:36.040253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b090b8-9ff5-4c23-b02c-6b69d573d848 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:36.042298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b090b8-9ff5-4c23-b02c-6b69d573d848 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:36.103461 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e32d15-558e-4c2a-af0c-36ebf8baec4d None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:36.103798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e32d15-558e-4c2a-af0c-36ebf8baec4d None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:36.104001 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e32d15-558e-4c2a-af0c-36ebf8baec4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:36.178811 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1115/2215] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:36 2022] POST /identity/v3/projects => generated 383 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:36.185080 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b090b8-9ff5-4c23-b02c-6b69d573d848 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:36.226026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-20e32d15-558e-4c2a-af0c-36ebf8baec4d None None] MFA Rules not processed for user `de28bb95bc464874a7f70b33e5c5f8e3`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:36.227371 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-20e32d15-558e-4c2a-af0c-36ebf8baec4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:36.333908 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b090b8-9ff5-4c23-b02c-6b69d573d848 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:36.346103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a277c82-b306-486b-9a3e-725816b96078 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:36.347396 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a277c82-b306-486b-9a3e-725816b96078 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:36.347882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a277c82-b306-486b-9a3e-725816b96078 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:36.347979 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a277c82-b306-486b-9a3e-725816b96078 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:36.348582 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a277c82-b306-486b-9a3e-725816b96078 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:36.350630 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a277c82-b306-486b-9a3e-725816b96078 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:36.376417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1101/2216] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:36 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:36.385951 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:36.391381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e32d15-558e-4c2a-af0c-36ebf8baec4d None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:36.392993 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-20e32d15-558e-4c2a-af0c-36ebf8baec4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:36.440376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1116/2217] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:36 2022] POST /identity/v3/users => generated 292 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:36.442858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a277c82-b306-486b-9a3e-725816b96078 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:36.445801 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-5a277c82-b306-486b-9a3e-725816b96078 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:36.468429 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a277c82-b306-486b-9a3e-725816b96078 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:36.474671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:36.482954 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78caa655-f3eb-4410-8f5a-38e804e7d009 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:36.485090 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78caa655-f3eb-4410-8f5a-38e804e7d009 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:36.485496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78caa655-f3eb-4410-8f5a-38e804e7d009 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:36.485873 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78caa655-f3eb-4410-8f5a-38e804e7d009 admin admin] PATH_INFO: `/v3/projects/da2bf84b8d644402b31dfe09fab21dab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:36.487522 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78caa655-f3eb-4410-8f5a-38e804e7d009 admin admin] RBAC: Authorizing `identity:delete_project(project_id=da2bf84b8d644402b31dfe09fab21dab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:36.492066 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78caa655-f3eb-4410-8f5a-38e804e7d009 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:36.602630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e32d15-558e-4c2a-af0c-36ebf8baec4d None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:36.609766 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e7d7ea-682b-4812-9bdd-666ae565195e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:36.611296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e7d7ea-682b-4812-9bdd-666ae565195e service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:36.611814 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e7d7ea-682b-4812-9bdd-666ae565195e service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:36.612313 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e7d7ea-682b-4812-9bdd-666ae565195e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:36.615970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-44e7d7ea-682b-4812-9bdd-666ae565195e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:36.681122 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-78caa655-f3eb-4410-8f5a-38e804e7d009 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'da2bf84b8d644402b31dfe09fab21dab', 'request_id': 'req-78caa655-f3eb-4410-8f5a-38e804e7d009'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:36.712703 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e7d7ea-682b-4812-9bdd-666ae565195e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:36.714538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e7d7ea-682b-4812-9bdd-666ae565195e service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:36.727203 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1102/2218] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:36 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:03:36.729512 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e7d7ea-682b-4812-9bdd-666ae565195e service nova] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:36.740371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-78caa655-f3eb-4410-8f5a-38e804e7d009 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'da2bf84b8d644402b31dfe09fab21dab', 'request_id': 'req-78caa655-f3eb-4410-8f5a-38e804e7d009'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:36.751265 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-78caa655-f3eb-4410-8f5a-38e804e7d009 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'da2bf84b8d644402b31dfe09fab21dab', 'request_id': 'req-78caa655-f3eb-4410-8f5a-38e804e7d009'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:36.764967 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e7d7ea-682b-4812-9bdd-666ae565195e service nova] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:36.779560 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-78caa655-f3eb-4410-8f5a-38e804e7d009 admin admin] The token cache is being invalidate because project da2bf84b8d644402b31dfe09fab21dab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:36.779859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-78caa655-f3eb-4410-8f5a-38e804e7d009 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:36.782139 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-78caa655-f3eb-4410-8f5a-38e804e7d009 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:36.799897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1117/2219] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:36 2022] DELETE /identity/v3/projects/da2bf84b8d644402b31dfe09fab21dab => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:36.805466 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0273aa0-8288-40f8-9da6-28dd29dba78a None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:36.805761 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0273aa0-8288-40f8-9da6-28dd29dba78a None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:36.806030 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0273aa0-8288-40f8-9da6-28dd29dba78a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:36.840683 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-d0273aa0-8288-40f8-9da6-28dd29dba78a None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:36.843948 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d0273aa0-8288-40f8-9da6-28dd29dba78a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:36.866773 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abd5598-0fae-481e-a1fe-664c0e779c28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:36.867563 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abd5598-0fae-481e-a1fe-664c0e779c28 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:36.868130 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abd5598-0fae-481e-a1fe-664c0e779c28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:36.868413 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5abd5598-0fae-481e-a1fe-664c0e779c28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:36.869105 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abd5598-0fae-481e-a1fe-664c0e779c28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:36.871571 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5abd5598-0fae-481e-a1fe-664c0e779c28 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:36.891210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1103/2220] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:36 2022] GET /identity/v3/roles => generated 1420 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:36.895451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abd5598-0fae-481e-a1fe-664c0e779c28 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:36.900333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-5abd5598-0fae-481e-a1fe-664c0e779c28 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:36.925810 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5abd5598-0fae-481e-a1fe-664c0e779c28 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:36.931801 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1118/2221] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:36.939648 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0273aa0-8288-40f8-9da6-28dd29dba78a None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:36.945079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d0273aa0-8288-40f8-9da6-28dd29dba78a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:36.949269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbab585-a2f1-417b-b5d8-704c653a8bb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:36.949905 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbab585-a2f1-417b-b5d8-704c653a8bb5 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:36.951632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbab585-a2f1-417b-b5d8-704c653a8bb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:36.952687 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbab585-a2f1-417b-b5d8-704c653a8bb5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:36.954912 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbab585-a2f1-417b-b5d8-704c653a8bb5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:36.957472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbab585-a2f1-417b-b5d8-704c653a8bb5 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:36.963884 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0273aa0-8288-40f8-9da6-28dd29dba78a None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:36.976545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1104/2222] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:36 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:36.979348 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83002ec6-75fd-4a34-a4d1-7973223c2d70 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:36.979873 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83002ec6-75fd-4a34-a4d1-7973223c2d70 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:36.980206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83002ec6-75fd-4a34-a4d1-7973223c2d70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:36.986517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a609090-5351-40b9-b284-04920a0b5c03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:36.989880 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a609090-5351-40b9-b284-04920a0b5c03 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:36.989880 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a609090-5351-40b9-b284-04920a0b5c03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:36.989880 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a609090-5351-40b9-b284-04920a0b5c03 admin admin] PATH_INFO: `/v3/users/88a8db8d7d494c62b65e427278ef9a2a` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:36.989880 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a609090-5351-40b9-b284-04920a0b5c03 admin admin] RBAC: Authorizing `identity:delete_user(user_id=88a8db8d7d494c62b65e427278ef9a2a)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:36.991498 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a609090-5351-40b9-b284-04920a0b5c03 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:37.012983 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-83002ec6-75fd-4a34-a4d1-7973223c2d70 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:37.015944 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-83002ec6-75fd-4a34-a4d1-7973223c2d70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:37.025139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1105/2223] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:37.026406 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83002ec6-75fd-4a34-a4d1-7973223c2d70 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:37.036330 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83002ec6-75fd-4a34-a4d1-7973223c2d70 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:37.044385 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84439d0d-bfee-46f3-8d42-803dd62e950c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:37.045147 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84439d0d-bfee-46f3-8d42-803dd62e950c admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:37.046534 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84439d0d-bfee-46f3-8d42-803dd62e950c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:37.047054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84439d0d-bfee-46f3-8d42-803dd62e950c admin admin] PATH_INFO: `/v3/projects/6b1d7115fcf643b28a49fc07d68ace6c/users/4b6de89c2fca496e8ce9cdac81eb6074/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:37.068626 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84439d0d-bfee-46f3-8d42-803dd62e950c admin admin] RBAC: Authorizing `identity:create_grant(project_id=6b1d7115fcf643b28a49fc07d68ace6c, user_id=4b6de89c2fca496e8ce9cdac81eb6074, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:37.071498 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84439d0d-bfee-46f3-8d42-803dd62e950c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:37.240700 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1106/2224] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:37 2022] PUT /identity/v3/projects/6b1d7115fcf643b28a49fc07d68ace6c/users/4b6de89c2fca496e8ce9cdac81eb6074/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:37.243552 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84439d0d-bfee-46f3-8d42-803dd62e950c admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:37.322747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84439d0d-bfee-46f3-8d42-803dd62e950c admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:37.331653 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d46fb8b-4414-44d6-acae-3d3c0786425e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:37.336744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d46fb8b-4414-44d6-acae-3d3c0786425e admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:37.336744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d46fb8b-4414-44d6-acae-3d3c0786425e admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:37.336744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d46fb8b-4414-44d6-acae-3d3c0786425e admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:37.336744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d46fb8b-4414-44d6-acae-3d3c0786425e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:03:37.336744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d46fb8b-4414-44d6-acae-3d3c0786425e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:37.336744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d46fb8b-4414-44d6-acae-3d3c0786425e admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:37.348917 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6a609090-5351-40b9-b284-04920a0b5c03 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '88a8db8d7d494c62b65e427278ef9a2a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:37.349325 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1107/2225] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:03:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:03:37.351162 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6573d62d-53a5-4d11-8150-51d17e94b55c None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:37.351451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6573d62d-53a5-4d11-8150-51d17e94b55c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:37.351729 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6573d62d-53a5-4d11-8150-51d17e94b55c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:37.389703 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-6573d62d-53a5-4d11-8150-51d17e94b55c None None] MFA Rules not processed for user `4b6de89c2fca496e8ce9cdac81eb6074`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:37.390991 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-6573d62d-53a5-4d11-8150-51d17e94b55c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:37.456703 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6a609090-5351-40b9-b284-04920a0b5c03 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '88a8db8d7d494c62b65e427278ef9a2a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:37.465705 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1108/2226] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:37 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:37.471102 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6573d62d-53a5-4d11-8150-51d17e94b55c None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:37.476014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6a609090-5351-40b9-b284-04920a0b5c03 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '88a8db8d7d494c62b65e427278ef9a2a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:37.476745 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6a609090-5351-40b9-b284-04920a0b5c03 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '88a8db8d7d494c62b65e427278ef9a2a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:37.485808 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:37.491821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6573d62d-53a5-4d11-8150-51d17e94b55c None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:37.498185 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f12cf3-21c0-4278-ae15-db446df8936b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:37.500424 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f12cf3-21c0-4278-ae15-db446df8936b admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:37.500715 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f12cf3-21c0-4278-ae15-db446df8936b admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:37.500993 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f12cf3-21c0-4278-ae15-db446df8936b admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:37.502392 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f12cf3-21c0-4278-ae15-db446df8936b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:37.505193 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f12cf3-21c0-4278-ae15-db446df8936b admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:37.505985 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6a609090-5351-40b9-b284-04920a0b5c03 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '88a8db8d7d494c62b65e427278ef9a2a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:37.524864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1119/2227] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:36 2022] DELETE /identity/v3/users/88a8db8d7d494c62b65e427278ef9a2a => generated 0 bytes in 589 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:37.524864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a609090-5351-40b9-b284-04920a0b5c03 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:37.525846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-6a609090-5351-40b9-b284-04920a0b5c03 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:37.560843 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:37.617162 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1109/2228] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:37 2022] POST /identity/v3/projects => generated 367 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:37.618798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f12cf3-21c0-4278-ae15-db446df8936b admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:37.625671 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-76f12cf3-21c0-4278-ae15-db446df8936b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:37.643752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a609090-5351-40b9-b284-04920a0b5c03 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:37.753809 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f12cf3-21c0-4278-ae15-db446df8936b admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:37.759189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb26b18f-c9bb-40df-a9ad-680a9e7c9e32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:37.760172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2831915e-fb5b-47e9-b539-e563368f9cce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:37.760815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2831915e-fb5b-47e9-b539-e563368f9cce admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:37.761097 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2831915e-fb5b-47e9-b539-e563368f9cce admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:37.761372 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2831915e-fb5b-47e9-b539-e563368f9cce admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:37.762014 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2831915e-fb5b-47e9-b539-e563368f9cce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:37.762482 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb26b18f-c9bb-40df-a9ad-680a9e7c9e32 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:37.762745 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb26b18f-c9bb-40df-a9ad-680a9e7c9e32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:37.763058 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb26b18f-c9bb-40df-a9ad-680a9e7c9e32 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:37.764075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2831915e-fb5b-47e9-b539-e563368f9cce admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:37.765192 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb26b18f-c9bb-40df-a9ad-680a9e7c9e32 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:37.769549 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb26b18f-c9bb-40df-a9ad-680a9e7c9e32 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:37.856128 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1110/2229] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:37 2022] POST /identity/v3/users => generated 284 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:37.861434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2831915e-fb5b-47e9-b539-e563368f9cce admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:37.880494 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2831915e-fb5b-47e9-b539-e563368f9cce admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:37.890251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d46ecea-ae5f-4fe1-bd94-67c7f8d4e261 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:37.891955 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d46ecea-ae5f-4fe1-bd94-67c7f8d4e261 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:37.892242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d46ecea-ae5f-4fe1-bd94-67c7f8d4e261 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:37.892657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d46ecea-ae5f-4fe1-bd94-67c7f8d4e261 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:37.893321 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d46ecea-ae5f-4fe1-bd94-67c7f8d4e261 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:37.894970 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d46ecea-ae5f-4fe1-bd94-67c7f8d4e261 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:37.910440 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1111/2230] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:37 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:03:37.910636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1120/2231] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:37 2022] POST /identity/v3/projects => generated 385 bytes in 387 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:37.911689 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d46ecea-ae5f-4fe1-bd94-67c7f8d4e261 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:37.914966 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb26b18f-c9bb-40df-a9ad-680a9e7c9e32 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:37.930508 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d46ecea-ae5f-4fe1-bd94-67c7f8d4e261 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:38.104816 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb26b18f-c9bb-40df-a9ad-680a9e7c9e32 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:38.115692 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d07520-9246-434b-968e-819633a4742d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:38.116310 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d07520-9246-434b-968e-819633a4742d admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:38.116588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d07520-9246-434b-968e-819633a4742d admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:38.116857 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d07520-9246-434b-968e-819633a4742d admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:38.117568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d07520-9246-434b-968e-819633a4742d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:38.117988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323b4fa3-b1b9-4fb5-b409-f3a423f9ab51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:38.119271 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323b4fa3-b1b9-4fb5-b409-f3a423f9ab51 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:38.119271 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323b4fa3-b1b9-4fb5-b409-f3a423f9ab51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:38.119477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d07520-9246-434b-968e-819633a4742d admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:38.119586 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323b4fa3-b1b9-4fb5-b409-f3a423f9ab51 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:38.119874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323b4fa3-b1b9-4fb5-b409-f3a423f9ab51 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:38.124879 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323b4fa3-b1b9-4fb5-b409-f3a423f9ab51 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:38.148468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1112/2232] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:37 2022] GET /identity/v3/roles => generated 1420 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:38.148468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d07520-9246-434b-968e-819633a4742d admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:38.149548 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a5d07520-9246-434b-968e-819633a4742d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:38.170684 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d07520-9246-434b-968e-819633a4742d admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:38.184975 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed0320f-f1bc-4498-a431-2fbc72f4b666 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:38.185666 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed0320f-f1bc-4498-a431-2fbc72f4b666 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:38.185943 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed0320f-f1bc-4498-a431-2fbc72f4b666 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:38.186800 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed0320f-f1bc-4498-a431-2fbc72f4b666 admin admin] PATH_INFO: `/v3/projects/8e1a67a82fb44cf59f9b4f30848881d1` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:38.187971 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed0320f-f1bc-4498-a431-2fbc72f4b666 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e1a67a82fb44cf59f9b4f30848881d1)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:38.190133 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed0320f-f1bc-4498-a431-2fbc72f4b666 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:38.232747 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1121/2233] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:37 2022] POST /identity/v3/users => generated 293 bytes in 318 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:38.233441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323b4fa3-b1b9-4fb5-b409-f3a423f9ab51 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:38.255181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323b4fa3-b1b9-4fb5-b409-f3a423f9ab51 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:38.260111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b19117-aeed-44c1-a3d9-ca9a0f10d45f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:38.260927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b19117-aeed-44c1-a3d9-ca9a0f10d45f admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:38.261317 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b19117-aeed-44c1-a3d9-ca9a0f10d45f admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:38.261646 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b19117-aeed-44c1-a3d9-ca9a0f10d45f admin admin] PATH_INFO: `/v3/projects/f045ee2237084f32a053b50c8299abff/users/843c85a2734e405a8649c9e5dcd2c183/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:38.278749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b19117-aeed-44c1-a3d9-ca9a0f10d45f admin admin] RBAC: Authorizing `identity:create_grant(project_id=f045ee2237084f32a053b50c8299abff, user_id=843c85a2734e405a8649c9e5dcd2c183, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:38.281572 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b19117-aeed-44c1-a3d9-ca9a0f10d45f admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:38.375647 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1122/2234] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:38 2022] PUT /identity/v3/projects/f045ee2237084f32a053b50c8299abff/users/843c85a2734e405a8649c9e5dcd2c183/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:38.376379 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b19117-aeed-44c1-a3d9-ca9a0f10d45f admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:38.390404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-8ed0320f-f1bc-4498-a431-2fbc72f4b666 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e1a67a82fb44cf59f9b4f30848881d1', 'request_id': 'req-8ed0320f-f1bc-4498-a431-2fbc72f4b666'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:38.437402 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-8ed0320f-f1bc-4498-a431-2fbc72f4b666 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e1a67a82fb44cf59f9b4f30848881d1', 'request_id': 'req-8ed0320f-f1bc-4498-a431-2fbc72f4b666'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:38.444005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-8ed0320f-f1bc-4498-a431-2fbc72f4b666 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e1a67a82fb44cf59f9b4f30848881d1', 'request_id': 'req-8ed0320f-f1bc-4498-a431-2fbc72f4b666'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:38.458813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b19117-aeed-44c1-a3d9-ca9a0f10d45f admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:38.465339 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-8ed0320f-f1bc-4498-a431-2fbc72f4b666 admin admin] The token cache is being invalidate because project 8e1a67a82fb44cf59f9b4f30848881d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:38.465339 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-8ed0320f-f1bc-4498-a431-2fbc72f4b666 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:38.465339 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-8ed0320f-f1bc-4498-a431-2fbc72f4b666 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:38.465339 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1113/2235] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:38 2022] DELETE /identity/v3/projects/8e1a67a82fb44cf59f9b4f30848881d1 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:38.471415 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5884d9ab-d2f5-4988-a1c2-4c97a9bebe04 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:38.471697 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5884d9ab-d2f5-4988-a1c2-4c97a9bebe04 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:38.471967 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5884d9ab-d2f5-4988-a1c2-4c97a9bebe04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:38.511131 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-5884d9ab-d2f5-4988-a1c2-4c97a9bebe04 None None] MFA Rules not processed for user `843c85a2734e405a8649c9e5dcd2c183`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:38.512524 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-5884d9ab-d2f5-4988-a1c2-4c97a9bebe04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:38.534993 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89503d8-ac23-469f-ab26-40ce948c7c10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:38.535880 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89503d8-ac23-469f-ab26-40ce948c7c10 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:38.536435 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89503d8-ac23-469f-ab26-40ce948c7c10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:38.536886 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89503d8-ac23-469f-ab26-40ce948c7c10 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:38.537692 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89503d8-ac23-469f-ab26-40ce948c7c10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:38.540214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a89503d8-ac23-469f-ab26-40ce948c7c10 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:38.554272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1123/2236] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:38 2022] GET /identity/v3/roles => generated 1420 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:38.554272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d88d01-c64a-46f5-8fb4-dd031850461d None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:38.554272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d88d01-c64a-46f5-8fb4-dd031850461d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:38.554272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d88d01-c64a-46f5-8fb4-dd031850461d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:38.573197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-86d88d01-c64a-46f5-8fb4-dd031850461d None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:38.574443 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-86d88d01-c64a-46f5-8fb4-dd031850461d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:38.582939 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1124/2237] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:38.589881 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d88d01-c64a-46f5-8fb4-dd031850461d None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:38.592275 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1114/2238] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:38 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:38.592275 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9eccd8-b50a-49d5-9846-9356860e78f2 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:38.592275 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9eccd8-b50a-49d5-9846-9356860e78f2 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:38.592275 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9eccd8-b50a-49d5-9846-9356860e78f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:38.595532 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-86d88d01-c64a-46f5-8fb4-dd031850461d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:38.608405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d88d01-c64a-46f5-8fb4-dd031850461d None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:38.612941 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-4e9eccd8-b50a-49d5-9846-9356860e78f2 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:38.613331 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88ef855-dbe2-4459-9f20-c41728428aac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:38.614035 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88ef855-dbe2-4459-9f20-c41728428aac admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:38.614289 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-4e9eccd8-b50a-49d5-9846-9356860e78f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:38.614487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88ef855-dbe2-4459-9f20-c41728428aac admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:38.620195 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88ef855-dbe2-4459-9f20-c41728428aac admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:38.620195 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88ef855-dbe2-4459-9f20-c41728428aac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:38.620195 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88ef855-dbe2-4459-9f20-c41728428aac admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:38.636377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1125/2240] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:38 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:38.636571 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1115/2240] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:38.637338 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88ef855-dbe2-4459-9f20-c41728428aac admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:38.642319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9eccd8-b50a-49d5-9846-9356860e78f2 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:38.648127 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c88ef855-dbe2-4459-9f20-c41728428aac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:38.652630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9eccd8-b50a-49d5-9846-9356860e78f2 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:38.659102 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b73a6a-f029-4abf-828d-3264cbed719f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:38.659361 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b73a6a-f029-4abf-828d-3264cbed719f admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:38.659623 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b73a6a-f029-4abf-828d-3264cbed719f admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:38.659911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b73a6a-f029-4abf-828d-3264cbed719f admin admin] PATH_INFO: `/v3/projects/662feae02a0c4b58afb4ce57482df1d2/users/6e47f07a139447fabc55bee0cff8909d/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:38.679491 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:38.681147 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b73a6a-f029-4abf-828d-3264cbed719f admin admin] RBAC: Authorizing `identity:create_grant(project_id=662feae02a0c4b58afb4ce57482df1d2, user_id=6e47f07a139447fabc55bee0cff8909d, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:38.685210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b73a6a-f029-4abf-828d-3264cbed719f admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:38.736134 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88ef855-dbe2-4459-9f20-c41728428aac admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:38.741501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8afe6c-d96d-44f0-aaa1-d114f005fca0 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'acdf4a7bd608492390176dcc42ca1267', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:38.741501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8afe6c-d96d-44f0-aaa1-d114f005fca0 service glance] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:38.741501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8afe6c-d96d-44f0-aaa1-d114f005fca0 service glance] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:38.741501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8afe6c-d96d-44f0-aaa1-d114f005fca0 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:38.751461 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8afe6c-d96d-44f0-aaa1-d114f005fca0 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:38.752934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8afe6c-d96d-44f0-aaa1-d114f005fca0 service glance] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:38.762105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1126/2241] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:38 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:38.763023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8afe6c-d96d-44f0-aaa1-d114f005fca0 service glance] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:38.776120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8afe6c-d96d-44f0-aaa1-d114f005fca0 service glance] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:38.784800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd990fd-2e6c-4c5f-9f76-2c9a8ae382bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:38.785611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd990fd-2e6c-4c5f-9f76-2c9a8ae382bf admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:38.785900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd990fd-2e6c-4c5f-9f76-2c9a8ae382bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:38.786477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd990fd-2e6c-4c5f-9f76-2c9a8ae382bf admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:38.789271 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd990fd-2e6c-4c5f-9f76-2c9a8ae382bf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:03:38.789271 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd990fd-2e6c-4c5f-9f76-2c9a8ae382bf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:38.800961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd990fd-2e6c-4c5f-9f76-2c9a8ae382bf admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:38.801384 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1116/2242] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:38 2022] PUT /identity/v3/projects/662feae02a0c4b58afb4ce57482df1d2/users/6e47f07a139447fabc55bee0cff8909d/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:38.803512 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980e0112-4879-414a-a3d1-fa682c484db4 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:38.803995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980e0112-4879-414a-a3d1-fa682c484db4 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:38.804426 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980e0112-4879-414a-a3d1-fa682c484db4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:38.809689 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1127/2243] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:03:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:03:38.819044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd990fd-2e6c-4c5f-9f76-2c9a8ae382bf admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:38.836793 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-980e0112-4879-414a-a3d1-fa682c484db4 None None] MFA Rules not processed for user `6e47f07a139447fabc55bee0cff8909d`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:38.837585 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd990fd-2e6c-4c5f-9f76-2c9a8ae382bf admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:38.838571 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-980e0112-4879-414a-a3d1-fa682c484db4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:38.940084 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:38.944222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1117/2244] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:38 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:38.948821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980e0112-4879-414a-a3d1-fa682c484db4 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:38.958868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee00e40-6971-4a52-8871-22d9ba4efd88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:38.959656 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee00e40-6971-4a52-8871-22d9ba4efd88 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:38.959859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee00e40-6971-4a52-8871-22d9ba4efd88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:38.960208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee00e40-6971-4a52-8871-22d9ba4efd88 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:38.960940 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee00e40-6971-4a52-8871-22d9ba4efd88 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:38.963713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee00e40-6971-4a52-8871-22d9ba4efd88 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:39.044740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980e0112-4879-414a-a3d1-fa682c484db4 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:39.056052 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9addd0f4-1d28-4561-8fb7-5597dd650f5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:39.058082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9addd0f4-1d28-4561-8fb7-5597dd650f5e admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:39.058653 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9addd0f4-1d28-4561-8fb7-5597dd650f5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:39.058888 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9addd0f4-1d28-4561-8fb7-5597dd650f5e admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:39.059737 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9addd0f4-1d28-4561-8fb7-5597dd650f5e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:39.064073 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9addd0f4-1d28-4561-8fb7-5597dd650f5e admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:39.069010 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1128/2245] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:38 2022] POST /identity/v3/projects => generated 389 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:03:39.073336 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee00e40-6971-4a52-8871-22d9ba4efd88 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:39.166399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:39.174210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1118/2246] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:38 2022] POST /identity/v3/projects => generated 385 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:39.178284 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9addd0f4-1d28-4561-8fb7-5597dd650f5e admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:39.183516 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee00e40-6971-4a52-8871-22d9ba4efd88 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:39.261992 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9addd0f4-1d28-4561-8fb7-5597dd650f5e admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:39.268179 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57598373-eb73-4f1c-a258-094e899a643e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:39.268947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd09a50d-580b-46b4-8e74-d2e3afdfe303 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:39.270002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd09a50d-580b-46b4-8e74-d2e3afdfe303 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:39.270278 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd09a50d-580b-46b4-8e74-d2e3afdfe303 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:39.270585 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd09a50d-580b-46b4-8e74-d2e3afdfe303 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:39.271258 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd09a50d-580b-46b4-8e74-d2e3afdfe303 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:39.273288 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd09a50d-580b-46b4-8e74-d2e3afdfe303 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:39.276814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57598373-eb73-4f1c-a258-094e899a643e admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:39.276814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57598373-eb73-4f1c-a258-094e899a643e admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:39.276814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57598373-eb73-4f1c-a258-094e899a643e admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:39.276814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57598373-eb73-4f1c-a258-094e899a643e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:39.277095 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57598373-eb73-4f1c-a258-094e899a643e admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:39.477479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1129/2247] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:39 2022] POST /identity/v3/users => generated 295 bytes in 404 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:39.481592 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57598373-eb73-4f1c-a258-094e899a643e admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:39.485760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1119/2248] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:39 2022] POST /identity/v3/users => generated 299 bytes in 305 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:03:39.488193 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd09a50d-580b-46b4-8e74-d2e3afdfe303 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:39.492712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57598373-eb73-4f1c-a258-094e899a643e admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:39.499997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffdb344-b712-422e-ba33-a272828fe66c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:39.506540 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffdb344-b712-422e-ba33-a272828fe66c admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:39.506540 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffdb344-b712-422e-ba33-a272828fe66c admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:39.506540 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffdb344-b712-422e-ba33-a272828fe66c admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:39.506540 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffdb344-b712-422e-ba33-a272828fe66c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:39.506540 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffdb344-b712-422e-ba33-a272828fe66c admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:39.508654 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd09a50d-580b-46b4-8e74-d2e3afdfe303 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:39.515814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1130/2249] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:39 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:03:39.515999 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fe54ba-df7d-44b5-bc54-fbd702a15594 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:39.516503 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fe54ba-df7d-44b5-bc54-fbd702a15594 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:39.516787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fe54ba-df7d-44b5-bc54-fbd702a15594 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:39.517060 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fe54ba-df7d-44b5-bc54-fbd702a15594 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:39.517742 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fe54ba-df7d-44b5-bc54-fbd702a15594 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:39.519926 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffdb344-b712-422e-ba33-a272828fe66c admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:39.520075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77fe54ba-df7d-44b5-bc54-fbd702a15594 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:39.550139 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffdb344-b712-422e-ba33-a272828fe66c admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:39.551997 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1120/2250] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:39 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:39.561995 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a039955-4483-4287-ae56-e5e5960066b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:39.565783 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a039955-4483-4287-ae56-e5e5960066b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:39.566069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fe54ba-df7d-44b5-bc54-fbd702a15594 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:39.566161 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a039955-4483-4287-ae56-e5e5960066b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:39.566161 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a039955-4483-4287-ae56-e5e5960066b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:39.566161 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a039955-4483-4287-ae56-e5e5960066b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:39.567410 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a039955-4483-4287-ae56-e5e5960066b1 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:39.576307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77fe54ba-df7d-44b5-bc54-fbd702a15594 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:39.586634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1131/2251] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:39 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:03:39.593440 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014b6154-13e5-46ba-9261-d1d72ce90cf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:39.594822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014b6154-13e5-46ba-9261-d1d72ce90cf5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:39.595005 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a039955-4483-4287-ae56-e5e5960066b1 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:39.595132 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014b6154-13e5-46ba-9261-d1d72ce90cf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:39.595502 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014b6154-13e5-46ba-9261-d1d72ce90cf5 admin admin] PATH_INFO: `/v3/projects/1fc3dc8ec8de4206a4c3a12dbb2a6e0d/users/b8887c97db054b3fa0ca6bf9c5d5db85/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:39.618319 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a039955-4483-4287-ae56-e5e5960066b1 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:39.632038 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b14be1-782f-497d-89ab-c6ea7f7d959d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:39.632192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014b6154-13e5-46ba-9261-d1d72ce90cf5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1fc3dc8ec8de4206a4c3a12dbb2a6e0d, user_id=b8887c97db054b3fa0ca6bf9c5d5db85, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:39.632778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b14be1-782f-497d-89ab-c6ea7f7d959d admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:39.633053 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b14be1-782f-497d-89ab-c6ea7f7d959d admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:39.633323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b14be1-782f-497d-89ab-c6ea7f7d959d admin admin] PATH_INFO: `/v3/projects/c21d140d48ba4014bc470e8cf720785d/users/d777c43fe5c142b3b5ea5e2485f4e49f/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:39.637983 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014b6154-13e5-46ba-9261-d1d72ce90cf5 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:39.660317 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b14be1-782f-497d-89ab-c6ea7f7d959d admin admin] RBAC: Authorizing `identity:create_grant(project_id=c21d140d48ba4014bc470e8cf720785d, user_id=d777c43fe5c142b3b5ea5e2485f4e49f, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:39.664296 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b14be1-782f-497d-89ab-c6ea7f7d959d admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:39.754579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1121/2252] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:39 2022] PUT /identity/v3/projects/1fc3dc8ec8de4206a4c3a12dbb2a6e0d/users/b8887c97db054b3fa0ca6bf9c5d5db85/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:39.758258 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014b6154-13e5-46ba-9261-d1d72ce90cf5 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:39.796242 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1132/2253] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:39 2022] PUT /identity/v3/projects/c21d140d48ba4014bc470e8cf720785d/users/d777c43fe5c142b3b5ea5e2485f4e49f/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:39.802559 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84728668-54fd-43e7-9705-5e6d95493a8c None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:39.804701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84728668-54fd-43e7-9705-5e6d95493a8c None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:39.804834 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84728668-54fd-43e7-9705-5e6d95493a8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:39.836910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-84728668-54fd-43e7-9705-5e6d95493a8c None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:39.838213 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-84728668-54fd-43e7-9705-5e6d95493a8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:39.869544 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014b6154-13e5-46ba-9261-d1d72ce90cf5 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:39.964014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1133/2254] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:40.004110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84728668-54fd-43e7-9705-5e6d95493a8c None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:40.004845 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0aa3f7-9e47-440b-9b6a-5ddcbaf6d09e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:40.005639 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0aa3f7-9e47-440b-9b6a-5ddcbaf6d09e admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:40.005922 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0aa3f7-9e47-440b-9b6a-5ddcbaf6d09e admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:40.006191 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0aa3f7-9e47-440b-9b6a-5ddcbaf6d09e admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:40.006977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0aa3f7-9e47-440b-9b6a-5ddcbaf6d09e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:40.007672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-84728668-54fd-43e7-9705-5e6d95493a8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:40.008587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0aa3f7-9e47-440b-9b6a-5ddcbaf6d09e admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:40.023963 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1122/2255] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:39 2022] GET /identity/v3/roles => generated 1420 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:03:40.034313 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0aa3f7-9e47-440b-9b6a-5ddcbaf6d09e admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:40.059473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0aa3f7-9e47-440b-9b6a-5ddcbaf6d09e admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:40.078968 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c490f752-8238-40c8-a1bc-99d6f3415a00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:40.079663 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c490f752-8238-40c8-a1bc-99d6f3415a00 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:40.080678 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c490f752-8238-40c8-a1bc-99d6f3415a00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:40.080678 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c490f752-8238-40c8-a1bc-99d6f3415a00 admin admin] PATH_INFO: `/v3/projects/1fc3dc8ec8de4206a4c3a12dbb2a6e0d/users/b8887c97db054b3fa0ca6bf9c5d5db85/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:40.085642 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c490f752-8238-40c8-a1bc-99d6f3415a00 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1fc3dc8ec8de4206a4c3a12dbb2a6e0d, user_id=b8887c97db054b3fa0ca6bf9c5d5db85, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:40.088881 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c490f752-8238-40c8-a1bc-99d6f3415a00 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:40.108878 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1123/2256] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:40 2022] PUT /identity/v3/projects/1fc3dc8ec8de4206a4c3a12dbb2a6e0d/users/b8887c97db054b3fa0ca6bf9c5d5db85/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:40.128951 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39446937-62f1-483b-8be8-15f553ee80ef None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:40.129247 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39446937-62f1-483b-8be8-15f553ee80ef None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:40.129526 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39446937-62f1-483b-8be8-15f553ee80ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:40.131154 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84728668-54fd-43e7-9705-5e6d95493a8c None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:40.216025 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-39446937-62f1-483b-8be8-15f553ee80ef None None] MFA Rules not processed for user `b8887c97db054b3fa0ca6bf9c5d5db85`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:40.218668 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-39446937-62f1-483b-8be8-15f553ee80ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:40.229469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd66ef54-7186-4a21-9438-2cbea63144d3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:40.230217 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd66ef54-7186-4a21-9438-2cbea63144d3 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:40.230589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd66ef54-7186-4a21-9438-2cbea63144d3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:40.230931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd66ef54-7186-4a21-9438-2cbea63144d3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:40.237782 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:40.358674 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1124/2257] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:40 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:40.378748 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:40.379956 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39446937-62f1-483b-8be8-15f553ee80ef None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:40.382032 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd66ef54-7186-4a21-9438-2cbea63144d3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:40.382959 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-39446937-62f1-483b-8be8-15f553ee80ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:40.383710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd66ef54-7186-4a21-9438-2cbea63144d3 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:40.398243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1134/2258] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:40.484201 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39446937-62f1-483b-8be8-15f553ee80ef None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:40.488244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19e274f-78a6-44d6-8560-b33f49345411 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:40.488808 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19e274f-78a6-44d6-8560-b33f49345411 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:40.489153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19e274f-78a6-44d6-8560-b33f49345411 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:40.494278 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19e274f-78a6-44d6-8560-b33f49345411 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:40.494278 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f19e274f-78a6-44d6-8560-b33f49345411 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:40.579724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19e274f-78a6-44d6-8560-b33f49345411 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:40.581564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19e274f-78a6-44d6-8560-b33f49345411 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:40.612038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1125/2259] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:40 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:41.252912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd66ef54-7186-4a21-9438-2cbea63144d3 service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:41.254316 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-bd66ef54-7186-4a21-9438-2cbea63144d3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:41.267554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd66ef54-7186-4a21-9438-2cbea63144d3 service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:41.273244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2bc252-1153-456c-bdee-381c1eda9b66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:41.273890 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2bc252-1153-456c-bdee-381c1eda9b66 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:41.274166 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2bc252-1153-456c-bdee-381c1eda9b66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:41.274524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2bc252-1153-456c-bdee-381c1eda9b66 admin admin] PATH_INFO: `/v3/users/4b7d120c2c0e455b92507aebaf7a31a1` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:41.275808 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2bc252-1153-456c-bdee-381c1eda9b66 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4b7d120c2c0e455b92507aebaf7a31a1)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:41.277888 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2bc252-1153-456c-bdee-381c1eda9b66 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:41.479527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8f2bc252-1153-456c-bdee-381c1eda9b66 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4b7d120c2c0e455b92507aebaf7a31a1'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:41.617905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8f2bc252-1153-456c-bdee-381c1eda9b66 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4b7d120c2c0e455b92507aebaf7a31a1'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:41.633326 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8f2bc252-1153-456c-bdee-381c1eda9b66 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4b7d120c2c0e455b92507aebaf7a31a1'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:41.634159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8f2bc252-1153-456c-bdee-381c1eda9b66 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4b7d120c2c0e455b92507aebaf7a31a1'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:41.646722 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8f2bc252-1153-456c-bdee-381c1eda9b66 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4b7d120c2c0e455b92507aebaf7a31a1'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:41.657671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1135/2260] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:41 2022] DELETE /identity/v3/users/4b7d120c2c0e455b92507aebaf7a31a1 => generated 0 bytes in 405 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:41.662619 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e25fc6f-14d4-4ba7-b977-eba52d0f7e6b None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:41.663125 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e25fc6f-14d4-4ba7-b977-eba52d0f7e6b None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:41.663202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e25fc6f-14d4-4ba7-b977-eba52d0f7e6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:41.679671 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:41.686564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-8e25fc6f-14d4-4ba7-b977-eba52d0f7e6b None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:41.687810 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8e25fc6f-14d4-4ba7-b977-eba52d0f7e6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:41.759944 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1126/2261] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:41.771305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2bc252-1153-456c-bdee-381c1eda9b66 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:41.773041 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8f2bc252-1153-456c-bdee-381c1eda9b66 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:41.861439 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2bc252-1153-456c-bdee-381c1eda9b66 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:41.863963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb960793-7c7d-4df8-9992-a7a43106a73c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:41.864777 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb960793-7c7d-4df8-9992-a7a43106a73c service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:41.865129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb960793-7c7d-4df8-9992-a7a43106a73c service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:41.865472 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb960793-7c7d-4df8-9992-a7a43106a73c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:41.877583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb960793-7c7d-4df8-9992-a7a43106a73c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:41.879818 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb960793-7c7d-4df8-9992-a7a43106a73c service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:41.891116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1136/2262] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:42.296026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e25fc6f-14d4-4ba7-b977-eba52d0f7e6b None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:42.301070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8e25fc6f-14d4-4ba7-b977-eba52d0f7e6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:42.314435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e25fc6f-14d4-4ba7-b977-eba52d0f7e6b None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:42.320318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f736c3-71b4-44a5-954c-f53fab909490 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:42.321262 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f736c3-71b4-44a5-954c-f53fab909490 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:42.321723 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f736c3-71b4-44a5-954c-f53fab909490 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:42.322189 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f736c3-71b4-44a5-954c-f53fab909490 admin admin] PATH_INFO: `/v3/projects/58e31a2ddf7741ac936bd5bd8eb40acb` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:42.323692 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f736c3-71b4-44a5-954c-f53fab909490 admin admin] RBAC: Authorizing `identity:delete_project(project_id=58e31a2ddf7741ac936bd5bd8eb40acb)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:42.327249 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f736c3-71b4-44a5-954c-f53fab909490 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:42.425336 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:42.456384 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-16f736c3-71b4-44a5-954c-f53fab909490 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '58e31a2ddf7741ac936bd5bd8eb40acb', 'request_id': 'req-16f736c3-71b4-44a5-954c-f53fab909490'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:42.528580 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-16f736c3-71b4-44a5-954c-f53fab909490 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '58e31a2ddf7741ac936bd5bd8eb40acb', 'request_id': 'req-16f736c3-71b4-44a5-954c-f53fab909490'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:42.542198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-16f736c3-71b4-44a5-954c-f53fab909490 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '58e31a2ddf7741ac936bd5bd8eb40acb', 'request_id': 'req-16f736c3-71b4-44a5-954c-f53fab909490'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:42.571587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-16f736c3-71b4-44a5-954c-f53fab909490 admin admin] The token cache is being invalidate because project 58e31a2ddf7741ac936bd5bd8eb40acb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:42.571882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-16f736c3-71b4-44a5-954c-f53fab909490 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:42.576055 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-16f736c3-71b4-44a5-954c-f53fab909490 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:42.582981 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1127/2263] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:42 2022] DELETE /identity/v3/projects/58e31a2ddf7741ac936bd5bd8eb40acb => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:42.888430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb960793-7c7d-4df8-9992-a7a43106a73c service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:42.890705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-eb960793-7c7d-4df8-9992-a7a43106a73c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:42.967804 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb960793-7c7d-4df8-9992-a7a43106a73c service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:43.029447 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea22ed4a-33ab-46f3-b510-8a45ee16cb22 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:43.030238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea22ed4a-33ab-46f3-b510-8a45ee16cb22 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:43.030642 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea22ed4a-33ab-46f3-b510-8a45ee16cb22 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:43.030918 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea22ed4a-33ab-46f3-b510-8a45ee16cb22 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:43.033127 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-ea22ed4a-33ab-46f3-b510-8a45ee16cb22 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:43.107185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea22ed4a-33ab-46f3-b510-8a45ee16cb22 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:43.109621 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea22ed4a-33ab-46f3-b510-8a45ee16cb22 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:43.125524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1137/2264] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:42 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:43.127235 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f736c3-71b4-44a5-954c-f53fab909490 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:43.128933 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-16f736c3-71b4-44a5-954c-f53fab909490 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:43.240085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f736c3-71b4-44a5-954c-f53fab909490 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:43.248864 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390fb303-7d09-4f5c-88b7-e0cbe2ba4c89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:43.257844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390fb303-7d09-4f5c-88b7-e0cbe2ba4c89 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:43.258559 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390fb303-7d09-4f5c-88b7-e0cbe2ba4c89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:43.258809 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390fb303-7d09-4f5c-88b7-e0cbe2ba4c89 admin admin] PATH_INFO: `/v3/users/35d18ce141664ccdbd9c0a1aac0af358` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:43.260313 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390fb303-7d09-4f5c-88b7-e0cbe2ba4c89 admin admin] RBAC: Authorizing `identity:delete_user(user_id=35d18ce141664ccdbd9c0a1aac0af358)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:43.262248 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390fb303-7d09-4f5c-88b7-e0cbe2ba4c89 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:43.367780 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:43.484504 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:43.501697 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-390fb303-7d09-4f5c-88b7-e0cbe2ba4c89 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '35d18ce141664ccdbd9c0a1aac0af358'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:43.571952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-390fb303-7d09-4f5c-88b7-e0cbe2ba4c89 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '35d18ce141664ccdbd9c0a1aac0af358'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:43.573008 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-390fb303-7d09-4f5c-88b7-e0cbe2ba4c89 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '35d18ce141664ccdbd9c0a1aac0af358'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:43.610003 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-390fb303-7d09-4f5c-88b7-e0cbe2ba4c89 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '35d18ce141664ccdbd9c0a1aac0af358'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:43.618788 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-390fb303-7d09-4f5c-88b7-e0cbe2ba4c89 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '35d18ce141664ccdbd9c0a1aac0af358'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:43.644569 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1128/2265] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:43 2022] DELETE /identity/v3/users/35d18ce141664ccdbd9c0a1aac0af358 => generated 0 bytes in 515 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:43.647234 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcfa72e-925d-4fca-bb2e-f317ab2f6d7c None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:43.648533 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcfa72e-925d-4fca-bb2e-f317ab2f6d7c None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:43.648533 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcfa72e-925d-4fca-bb2e-f317ab2f6d7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:43.692363 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-7fcfa72e-925d-4fca-bb2e-f317ab2f6d7c None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:43.693791 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-7fcfa72e-925d-4fca-bb2e-f317ab2f6d7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:43.786043 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1138/2266] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:43.801786 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390fb303-7d09-4f5c-88b7-e0cbe2ba4c89 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:43.803489 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-390fb303-7d09-4f5c-88b7-e0cbe2ba4c89 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:43.909410 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390fb303-7d09-4f5c-88b7-e0cbe2ba4c89 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:43.923185 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc52a6f-9677-4ec3-ae19-108881260821 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:43.927170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc52a6f-9677-4ec3-ae19-108881260821 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:43.927170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc52a6f-9677-4ec3-ae19-108881260821 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:43.927451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc52a6f-9677-4ec3-ae19-108881260821 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:43.948685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc52a6f-9677-4ec3-ae19-108881260821 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:43.950147 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc52a6f-9677-4ec3-ae19-108881260821 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:43.964773 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1129/2267] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:44.550880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:44.656269 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:44.801882 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcfa72e-925d-4fca-bb2e-f317ab2f6d7c None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:44.803454 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-7fcfa72e-925d-4fca-bb2e-f317ab2f6d7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:44.827366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcfa72e-925d-4fca-bb2e-f317ab2f6d7c None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:44.833580 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ced4fe-4b82-45f6-8b96-4b3620657c27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:44.834319 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ced4fe-4b82-45f6-8b96-4b3620657c27 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:44.835111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ced4fe-4b82-45f6-8b96-4b3620657c27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:44.835397 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ced4fe-4b82-45f6-8b96-4b3620657c27 admin admin] PATH_INFO: `/v3/projects/15cacfd8a7114dc891d48c3e30df3733` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:44.836645 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ced4fe-4b82-45f6-8b96-4b3620657c27 admin admin] RBAC: Authorizing `identity:delete_project(project_id=15cacfd8a7114dc891d48c3e30df3733)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:44.839192 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10ced4fe-4b82-45f6-8b96-4b3620657c27 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:44.996800 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc52a6f-9677-4ec3-ae19-108881260821 service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:44.999224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2dc52a6f-9677-4ec3-ae19-108881260821 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:45.030246 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-10ced4fe-4b82-45f6-8b96-4b3620657c27 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '15cacfd8a7114dc891d48c3e30df3733', 'request_id': 'req-10ced4fe-4b82-45f6-8b96-4b3620657c27'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:45.099385 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc52a6f-9677-4ec3-ae19-108881260821 service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:45.163249 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02ea191-1fde-4df3-b3e4-f15eb2b7978e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:45.164161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02ea191-1fde-4df3-b3e4-f15eb2b7978e service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:45.164438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02ea191-1fde-4df3-b3e4-f15eb2b7978e service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:45.164711 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02ea191-1fde-4df3-b3e4-f15eb2b7978e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:45.166527 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-c02ea191-1fde-4df3-b3e4-f15eb2b7978e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:45.174288 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-10ced4fe-4b82-45f6-8b96-4b3620657c27 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '15cacfd8a7114dc891d48c3e30df3733', 'request_id': 'req-10ced4fe-4b82-45f6-8b96-4b3620657c27'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:45.193766 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-10ced4fe-4b82-45f6-8b96-4b3620657c27 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '15cacfd8a7114dc891d48c3e30df3733', 'request_id': 'req-10ced4fe-4b82-45f6-8b96-4b3620657c27'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:45.219345 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-10ced4fe-4b82-45f6-8b96-4b3620657c27 admin admin] The token cache is being invalidate because project 15cacfd8a7114dc891d48c3e30df3733 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:45.222683 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-10ced4fe-4b82-45f6-8b96-4b3620657c27 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:45.224819 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-10ced4fe-4b82-45f6-8b96-4b3620657c27 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:45.236804 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1139/2268] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:44 2022] DELETE /identity/v3/projects/15cacfd8a7114dc891d48c3e30df3733 => generated 0 bytes in 432 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:45.238221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a74644-f93d-4db6-b7d4-20c65a878565 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:45.238795 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a74644-f93d-4db6-b7d4-20c65a878565 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:45.239080 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a74644-f93d-4db6-b7d4-20c65a878565 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:45.286206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-59a74644-f93d-4db6-b7d4-20c65a878565 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:45.287715 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-59a74644-f93d-4db6-b7d4-20c65a878565 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:45.298979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02ea191-1fde-4df3-b3e4-f15eb2b7978e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:45.302671 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02ea191-1fde-4df3-b3e4-f15eb2b7978e service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:45.330680 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1130/2269] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:44 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:03:45.381190 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1140/2270] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:45.386278 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8089f954-49da-49a0-8d4c-e12896367fe2 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:45.386614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8089f954-49da-49a0-8d4c-e12896367fe2 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:45.386859 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8089f954-49da-49a0-8d4c-e12896367fe2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:45.413903 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-8089f954-49da-49a0-8d4c-e12896367fe2 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:45.415077 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8089f954-49da-49a0-8d4c-e12896367fe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:45.424162 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1131/2271] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:45.428795 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a74644-f93d-4db6-b7d4-20c65a878565 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:45.442463 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a74644-f93d-4db6-b7d4-20c65a878565 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:45.454030 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a765bb01-c9c3-4e21-81cf-f001d386edf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:45.454809 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a765bb01-c9c3-4e21-81cf-f001d386edf7 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:45.455167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a765bb01-c9c3-4e21-81cf-f001d386edf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:45.455444 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a765bb01-c9c3-4e21-81cf-f001d386edf7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:45.456106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a765bb01-c9c3-4e21-81cf-f001d386edf7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:03:45.456408 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a765bb01-c9c3-4e21-81cf-f001d386edf7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:45.461049 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a765bb01-c9c3-4e21-81cf-f001d386edf7 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:45.471826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1141/2272] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:03:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:03:45.476308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8089f954-49da-49a0-8d4c-e12896367fe2 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:45.485765 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8089f954-49da-49a0-8d4c-e12896367fe2 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:45.491634 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328b06c8-c0bc-4a17-8427-c5cbb6e918d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:45.492325 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328b06c8-c0bc-4a17-8427-c5cbb6e918d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:45.492644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328b06c8-c0bc-4a17-8427-c5cbb6e918d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:45.492923 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328b06c8-c0bc-4a17-8427-c5cbb6e918d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:45.493574 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328b06c8-c0bc-4a17-8427-c5cbb6e918d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:45.496188 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328b06c8-c0bc-4a17-8427-c5cbb6e918d9 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:45.584441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1132/2273] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:45 2022] POST /identity/v3/projects => generated 375 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:03:45.589220 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a765bb01-c9c3-4e21-81cf-f001d386edf7 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:45.607987 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a765bb01-c9c3-4e21-81cf-f001d386edf7 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:45.697236 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31aa46e4-78cb-440b-9309-5de21b2a25b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:45.698297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aa46e4-78cb-440b-9309-5de21b2a25b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:45.698297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aa46e4-78cb-440b-9309-5de21b2a25b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:45.698885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aa46e4-78cb-440b-9309-5de21b2a25b2 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:45.699622 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31aa46e4-78cb-440b-9309-5de21b2a25b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:45.702529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31aa46e4-78cb-440b-9309-5de21b2a25b2 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:45.842250 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1142/2274] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:45 2022] POST /identity/v3/users => generated 288 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:45.846825 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328b06c8-c0bc-4a17-8427-c5cbb6e918d9 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:45.857551 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328b06c8-c0bc-4a17-8427-c5cbb6e918d9 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:45.860338 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:45.868268 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089f90d3-9ed5-4923-a041-bd6e19f3e322 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:45.869796 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089f90d3-9ed5-4923-a041-bd6e19f3e322 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:45.870569 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089f90d3-9ed5-4923-a041-bd6e19f3e322 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:45.870569 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089f90d3-9ed5-4923-a041-bd6e19f3e322 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:45.872589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089f90d3-9ed5-4923-a041-bd6e19f3e322 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:45.879034 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089f90d3-9ed5-4923-a041-bd6e19f3e322 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:45.891133 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1133/2275] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:45 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:45.895842 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31aa46e4-78cb-440b-9309-5de21b2a25b2 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:45.906701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31aa46e4-78cb-440b-9309-5de21b2a25b2 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:45.912071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaaf46f3-baa0-43f7-90e5-05ee2b459c57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:45.912771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaf46f3-baa0-43f7-90e5-05ee2b459c57 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:45.913189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaf46f3-baa0-43f7-90e5-05ee2b459c57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:45.913319 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaf46f3-baa0-43f7-90e5-05ee2b459c57 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:45.913974 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaaf46f3-baa0-43f7-90e5-05ee2b459c57 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:45.916684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaaf46f3-baa0-43f7-90e5-05ee2b459c57 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:45.935876 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089f90d3-9ed5-4923-a041-bd6e19f3e322 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:45.943150 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1143/2276] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:45 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:45.951880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089f90d3-9ed5-4923-a041-bd6e19f3e322 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:45.961313 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95dbfc3-5137-4ebc-b4bd-6829012e315f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:45.962069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95dbfc3-5137-4ebc-b4bd-6829012e315f admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:45.962397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95dbfc3-5137-4ebc-b4bd-6829012e315f admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:45.962677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95dbfc3-5137-4ebc-b4bd-6829012e315f admin admin] PATH_INFO: `/v3/projects/cf3422e0aa2840b6b15c1b48f86c476c/users/c0a716efec294c4593fb9eb865c27afd/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:45.993082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95dbfc3-5137-4ebc-b4bd-6829012e315f admin admin] RBAC: Authorizing `identity:create_grant(project_id=cf3422e0aa2840b6b15c1b48f86c476c, user_id=c0a716efec294c4593fb9eb865c27afd, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:45.997156 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95dbfc3-5137-4ebc-b4bd-6829012e315f admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:46.117346 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1134/2277] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:45 2022] PUT /identity/v3/projects/cf3422e0aa2840b6b15c1b48f86c476c/users/c0a716efec294c4593fb9eb865c27afd/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:46.129339 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a249a36-a5a8-4e47-b739-8942e12d95a2 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:46.129621 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a249a36-a5a8-4e47-b739-8942e12d95a2 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:46.129889 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a249a36-a5a8-4e47-b739-8942e12d95a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:46.162767 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-1a249a36-a5a8-4e47-b739-8942e12d95a2 None None] MFA Rules not processed for user `c0a716efec294c4593fb9eb865c27afd`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:46.164143 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-1a249a36-a5a8-4e47-b739-8942e12d95a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:46.277401 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1144/2278] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:46 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:46.278525 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95dbfc3-5137-4ebc-b4bd-6829012e315f admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:46.366153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95dbfc3-5137-4ebc-b4bd-6829012e315f admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:46.374761 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c15e34-800f-4afd-ab68-a882015b3404 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:46.375643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c15e34-800f-4afd-ab68-a882015b3404 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:46.375988 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c15e34-800f-4afd-ab68-a882015b3404 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:46.376318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c15e34-800f-4afd-ab68-a882015b3404 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:46.377151 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c15e34-800f-4afd-ab68-a882015b3404 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:46.379578 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c15e34-800f-4afd-ab68-a882015b3404 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:46.385870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:46.474420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1135/2279] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:46 2022] POST /identity/v3/projects => generated 373 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:46.479039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a249a36-a5a8-4e47-b739-8942e12d95a2 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:46.497300 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a249a36-a5a8-4e47-b739-8942e12d95a2 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:46.553476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:46.572721 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d82bbcc-bee9-4948-8c4b-bcb79d722dc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:46.573427 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d82bbcc-bee9-4948-8c4b-bcb79d722dc8 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:46.573771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d82bbcc-bee9-4948-8c4b-bcb79d722dc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:46.574106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d82bbcc-bee9-4948-8c4b-bcb79d722dc8 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:46.574845 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d82bbcc-bee9-4948-8c4b-bcb79d722dc8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:46.576903 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d82bbcc-bee9-4948-8c4b-bcb79d722dc8 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:46.713220 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c15e34-800f-4afd-ab68-a882015b3404 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:46.719411 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1145/2280] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:46 2022] POST /identity/v3/users => generated 293 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:03:46.731860 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c15e34-800f-4afd-ab68-a882015b3404 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:46.741381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e48afbf-4d4e-4982-8503-cbaf499867f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:46.742177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e48afbf-4d4e-4982-8503-cbaf499867f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:46.742557 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e48afbf-4d4e-4982-8503-cbaf499867f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:46.744245 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e48afbf-4d4e-4982-8503-cbaf499867f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:46.744245 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e48afbf-4d4e-4982-8503-cbaf499867f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:46.745242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e48afbf-4d4e-4982-8503-cbaf499867f0 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:46.767011 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d82bbcc-bee9-4948-8c4b-bcb79d722dc8 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:46.775145 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1136/2281] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:46 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:46.785657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d82bbcc-bee9-4948-8c4b-bcb79d722dc8 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:46.793475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42f7468-5f7c-4c7d-b07b-ac7edd2e5da0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:46.795401 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42f7468-5f7c-4c7d-b07b-ac7edd2e5da0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:46.795735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42f7468-5f7c-4c7d-b07b-ac7edd2e5da0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:46.797311 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42f7468-5f7c-4c7d-b07b-ac7edd2e5da0 admin admin] PATH_INFO: `/v3/projects/aa0d5c5b30164752bd874e3568d19891/users/2c11737581e443fd81a439148fd4d0f0/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:46.825433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42f7468-5f7c-4c7d-b07b-ac7edd2e5da0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa0d5c5b30164752bd874e3568d19891, user_id=2c11737581e443fd81a439148fd4d0f0, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:46.829584 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42f7468-5f7c-4c7d-b07b-ac7edd2e5da0 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:46.927420 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1146/2282] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:46 2022] PUT /identity/v3/projects/aa0d5c5b30164752bd874e3568d19891/users/2c11737581e443fd81a439148fd4d0f0/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:46.931967 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e48afbf-4d4e-4982-8503-cbaf499867f0 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:46.964118 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e48afbf-4d4e-4982-8503-cbaf499867f0 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:47.100499 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-124a855b-c141-40ea-9761-c60caf93fe7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:47.101560 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124a855b-c141-40ea-9761-c60caf93fe7c admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:47.101988 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124a855b-c141-40ea-9761-c60caf93fe7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:47.102624 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-124a855b-c141-40ea-9761-c60caf93fe7c admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:47.103510 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124a855b-c141-40ea-9761-c60caf93fe7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:47.105695 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-124a855b-c141-40ea-9761-c60caf93fe7c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:47.124486 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1137/2283] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:46 2022] GET /identity/v3/roles => generated 1420 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:47.125111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42f7468-5f7c-4c7d-b07b-ac7edd2e5da0 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:47.145859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42f7468-5f7c-4c7d-b07b-ac7edd2e5da0 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:47.157264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7ca9b8-4b20-489e-9e7f-ccd44dca2b6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:47.157859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7ca9b8-4b20-489e-9e7f-ccd44dca2b6f admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:47.161684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7ca9b8-4b20-489e-9e7f-ccd44dca2b6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:47.161684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7ca9b8-4b20-489e-9e7f-ccd44dca2b6f admin admin] PATH_INFO: `/v3/projects/aa0d5c5b30164752bd874e3568d19891/users/2c11737581e443fd81a439148fd4d0f0/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:47.161684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7ca9b8-4b20-489e-9e7f-ccd44dca2b6f admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa0d5c5b30164752bd874e3568d19891, user_id=2c11737581e443fd81a439148fd4d0f0, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:47.164966 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7ca9b8-4b20-489e-9e7f-ccd44dca2b6f admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:47.180043 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1147/2284] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:47 2022] PUT /identity/v3/projects/aa0d5c5b30164752bd874e3568d19891/users/2c11737581e443fd81a439148fd4d0f0/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:47.192911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d7de69-25c9-4872-b717-3f3b4ae13de5 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:47.193288 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d7de69-25c9-4872-b717-3f3b4ae13de5 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:47.193667 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d7de69-25c9-4872-b717-3f3b4ae13de5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:47.227088 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-39d7de69-25c9-4872-b717-3f3b4ae13de5 None None] MFA Rules not processed for user `2c11737581e443fd81a439148fd4d0f0`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:47.228599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-39d7de69-25c9-4872-b717-3f3b4ae13de5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:47.239633 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6b177e-bfa6-45cb-997f-01f85a297833 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:47.240144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6b177e-bfa6-45cb-997f-01f85a297833 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:47.240819 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6b177e-bfa6-45cb-997f-01f85a297833 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:47.286712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-8b6b177e-bfa6-45cb-997f-01f85a297833 None None] MFA Rules not processed for user `d777c43fe5c142b3b5ea5e2485f4e49f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:47.288128 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8b6b177e-bfa6-45cb-997f-01f85a297833 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:47.367250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1138/2285] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:47 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:47.376458 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d7de69-25c9-4872-b717-3f3b4ae13de5 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:47.379300 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-39d7de69-25c9-4872-b717-3f3b4ae13de5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:47.427589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1148/2286] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:47 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:47.456652 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6b177e-bfa6-45cb-997f-01f85a297833 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:47.457990 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d7de69-25c9-4872-b717-3f3b4ae13de5 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:47.459362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8b6b177e-bfa6-45cb-997f-01f85a297833 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:47.462057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2626da8b-dd54-4561-8d6e-ae961b3319ff service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:47.462877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2626da8b-dd54-4561-8d6e-ae961b3319ff service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:47.463289 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2626da8b-dd54-4561-8d6e-ae961b3319ff service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:47.463629 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2626da8b-dd54-4561-8d6e-ae961b3319ff service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:47.546908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2626da8b-dd54-4561-8d6e-ae961b3319ff service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:47.548573 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2626da8b-dd54-4561-8d6e-ae961b3319ff service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:47.566410 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1139/2287] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:47 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:47.581074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6b177e-bfa6-45cb-997f-01f85a297833 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:47.597152 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4034f17-599d-4da5-820a-8083363ce258 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:47.597784 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4034f17-599d-4da5-820a-8083363ce258 service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:47.598064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4034f17-599d-4da5-820a-8083363ce258 service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:47.598546 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4034f17-599d-4da5-820a-8083363ce258 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:47.625018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4034f17-599d-4da5-820a-8083363ce258 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:47.626152 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4034f17-599d-4da5-820a-8083363ce258 service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:47.661553 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1149/2288] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:47 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:47.677386 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:48.097876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:49.271048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:49.728188 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=79,keystone:INSERT=5,keystone:DELETE=16 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:50.828625 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2626da8b-dd54-4561-8d6e-ae961b3319ff service cinder] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:50.830072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2626da8b-dd54-4561-8d6e-ae961b3319ff service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:50.911173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2626da8b-dd54-4561-8d6e-ae961b3319ff service cinder] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:50.917539 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5423125d-29d2-4349-b7b0-8281724c79ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:50.918239 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5423125d-29d2-4349-b7b0-8281724c79ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:50.919811 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5423125d-29d2-4349-b7b0-8281724c79ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:50.920203 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5423125d-29d2-4349-b7b0-8281724c79ab admin admin] PATH_INFO: `/v3/users/4b6de89c2fca496e8ce9cdac81eb6074` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:50.921446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5423125d-29d2-4349-b7b0-8281724c79ab admin admin] RBAC: Authorizing `identity:delete_user(user_id=4b6de89c2fca496e8ce9cdac81eb6074)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:50.923780 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5423125d-29d2-4349-b7b0-8281724c79ab admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:50.945648 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c955007a-19f5-4221-8eb2-cfc52f021966 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:50.946554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c955007a-19f5-4221-8eb2-cfc52f021966 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:50.947032 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c955007a-19f5-4221-8eb2-cfc52f021966 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:50.956298 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:50.970832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c955007a-19f5-4221-8eb2-cfc52f021966 None None] MFA Rules not processed for user `d777c43fe5c142b3b5ea5e2485f4e49f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:50.972062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c955007a-19f5-4221-8eb2-cfc52f021966 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:50.979340 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1150/2289] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:50 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:51.121027 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5423125d-29d2-4349-b7b0-8281724c79ab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4b6de89c2fca496e8ce9cdac81eb6074'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:51.138278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=95,keystone:DELETE=1,keystone:INSERT=4 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:51.219729 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5423125d-29d2-4349-b7b0-8281724c79ab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4b6de89c2fca496e8ce9cdac81eb6074'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:51.222873 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5423125d-29d2-4349-b7b0-8281724c79ab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4b6de89c2fca496e8ce9cdac81eb6074'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:51.244364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5423125d-29d2-4349-b7b0-8281724c79ab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4b6de89c2fca496e8ce9cdac81eb6074'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:51.254863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5423125d-29d2-4349-b7b0-8281724c79ab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4b6de89c2fca496e8ce9cdac81eb6074'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:51.265862 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1140/2290] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:50 2022] DELETE /identity/v3/users/4b6de89c2fca496e8ce9cdac81eb6074 => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:51.270254 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faec9477-cdac-419f-a757-9f2a003f128e None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:51.270663 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faec9477-cdac-419f-a757-9f2a003f128e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:51.270998 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faec9477-cdac-419f-a757-9f2a003f128e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:51.313045 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-faec9477-cdac-419f-a757-9f2a003f128e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:51.314298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-faec9477-cdac-419f-a757-9f2a003f128e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:51.347231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5423125d-29d2-4349-b7b0-8281724c79ab admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:51.348640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-5423125d-29d2-4349-b7b0-8281724c79ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:51.397521 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1151/2291] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:51.408812 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faec9477-cdac-419f-a757-9f2a003f128e None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:51.410540 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-faec9477-cdac-419f-a757-9f2a003f128e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:51.502460 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5423125d-29d2-4349-b7b0-8281724c79ab admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:51.516949 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450c8d61-385b-4c41-bbb5-f6ac432fbd2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:51.518309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450c8d61-385b-4c41-bbb5-f6ac432fbd2f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:51.518309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450c8d61-385b-4c41-bbb5-f6ac432fbd2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:51.518309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450c8d61-385b-4c41-bbb5-f6ac432fbd2f admin admin] PATH_INFO: `/v3/users/c0a716efec294c4593fb9eb865c27afd` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:51.520582 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faec9477-cdac-419f-a757-9f2a003f128e None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:51.521689 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450c8d61-385b-4c41-bbb5-f6ac432fbd2f admin admin] RBAC: Authorizing `identity:delete_user(user_id=c0a716efec294c4593fb9eb865c27afd)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:51.523954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450c8d61-385b-4c41-bbb5-f6ac432fbd2f admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:51.526862 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8720b630-8968-407e-8735-bc60d28dda64 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:51.527667 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8720b630-8968-407e-8735-bc60d28dda64 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:51.527989 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8720b630-8968-407e-8735-bc60d28dda64 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:51.528211 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8720b630-8968-407e-8735-bc60d28dda64 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:51.539622 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8720b630-8968-407e-8735-bc60d28dda64 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:51.548902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8720b630-8968-407e-8735-bc60d28dda64 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:51.554499 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1152/2292] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:51.748124 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-450c8d61-385b-4c41-bbb5-f6ac432fbd2f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c0a716efec294c4593fb9eb865c27afd'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:51.847470 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-450c8d61-385b-4c41-bbb5-f6ac432fbd2f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c0a716efec294c4593fb9eb865c27afd'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:51.848621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-450c8d61-385b-4c41-bbb5-f6ac432fbd2f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c0a716efec294c4593fb9eb865c27afd'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:51.864999 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-450c8d61-385b-4c41-bbb5-f6ac432fbd2f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c0a716efec294c4593fb9eb865c27afd'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:51.874284 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-450c8d61-385b-4c41-bbb5-f6ac432fbd2f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c0a716efec294c4593fb9eb865c27afd'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:51.882188 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1141/2293] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:51 2022] DELETE /identity/v3/users/c0a716efec294c4593fb9eb865c27afd => generated 0 bytes in 536 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:51.887060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99a7a05-7c13-4148-bfb4-7d7f890b8652 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:51.887507 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99a7a05-7c13-4148-bfb4-7d7f890b8652 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:51.893730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99a7a05-7c13-4148-bfb4-7d7f890b8652 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:51.917431 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-a99a7a05-7c13-4148-bfb4-7d7f890b8652 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:51.918855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a99a7a05-7c13-4148-bfb4-7d7f890b8652 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:51.984427 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1153/2294] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:51.994230 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450c8d61-385b-4c41-bbb5-f6ac432fbd2f admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:51.999064 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-450c8d61-385b-4c41-bbb5-f6ac432fbd2f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:52.020244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:52.055059 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450c8d61-385b-4c41-bbb5-f6ac432fbd2f admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:52.058614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cca4e60-8caa-43de-8f39-ec9cb18c7c9c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:52.059211 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cca4e60-8caa-43de-8f39-ec9cb18c7c9c service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:52.061965 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cca4e60-8caa-43de-8f39-ec9cb18c7c9c service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:52.061965 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cca4e60-8caa-43de-8f39-ec9cb18c7c9c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:52.069504 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cca4e60-8caa-43de-8f39-ec9cb18c7c9c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:52.070956 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cca4e60-8caa-43de-8f39-ec9cb18c7c9c service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:52.080682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1142/2295] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:52.088485 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99a7a05-7c13-4148-bfb4-7d7f890b8652 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:52.089866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a99a7a05-7c13-4148-bfb4-7d7f890b8652 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:52.103879 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99a7a05-7c13-4148-bfb4-7d7f890b8652 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:52.109106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf0bb57-58c2-4887-8259-88d6f4238334 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:52.110018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf0bb57-58c2-4887-8259-88d6f4238334 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:52.110201 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf0bb57-58c2-4887-8259-88d6f4238334 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:52.110670 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf0bb57-58c2-4887-8259-88d6f4238334 admin admin] PATH_INFO: `/v3/projects/6b1d7115fcf643b28a49fc07d68ace6c` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:52.111949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf0bb57-58c2-4887-8259-88d6f4238334 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6b1d7115fcf643b28a49fc07d68ace6c)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:52.114038 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf0bb57-58c2-4887-8259-88d6f4238334 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:52.312012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fdf0bb57-58c2-4887-8259-88d6f4238334 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6b1d7115fcf643b28a49fc07d68ace6c', 'request_id': 'req-fdf0bb57-58c2-4887-8259-88d6f4238334'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:52.430099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fdf0bb57-58c2-4887-8259-88d6f4238334 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6b1d7115fcf643b28a49fc07d68ace6c', 'request_id': 'req-fdf0bb57-58c2-4887-8259-88d6f4238334'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:52.436378 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fdf0bb57-58c2-4887-8259-88d6f4238334 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6b1d7115fcf643b28a49fc07d68ace6c', 'request_id': 'req-fdf0bb57-58c2-4887-8259-88d6f4238334'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:52.447792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fdf0bb57-58c2-4887-8259-88d6f4238334 admin admin] The token cache is being invalidate because project 6b1d7115fcf643b28a49fc07d68ace6c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:52.448154 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fdf0bb57-58c2-4887-8259-88d6f4238334 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:52.448901 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fdf0bb57-58c2-4887-8259-88d6f4238334 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:52.450640 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1154/2296] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:52 2022] DELETE /identity/v3/projects/6b1d7115fcf643b28a49fc07d68ace6c => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:52.454864 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cca4e60-8caa-43de-8f39-ec9cb18c7c9c service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:52.458881 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:52.466477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cca4e60-8caa-43de-8f39-ec9cb18c7c9c service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:52.527244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f015424-f029-467b-b2a9-5f3ed861328d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:52.527826 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f015424-f029-467b-b2a9-5f3ed861328d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:52.528104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f015424-f029-467b-b2a9-5f3ed861328d admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:52.528984 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f015424-f029-467b-b2a9-5f3ed861328d admin admin] PATH_INFO: `/v3/users/6e47f07a139447fabc55bee0cff8909d` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:52.530158 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f015424-f029-467b-b2a9-5f3ed861328d admin admin] RBAC: Authorizing `identity:delete_user(user_id=6e47f07a139447fabc55bee0cff8909d)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:52.532123 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f015424-f029-467b-b2a9-5f3ed861328d admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:52.615475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf0bb57-58c2-4887-8259-88d6f4238334 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:52.617111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-fdf0bb57-58c2-4887-8259-88d6f4238334 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:52.630505 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf0bb57-58c2-4887-8259-88d6f4238334 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:52.635639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea2a43d-764c-4c4d-9985-0347ce8f667b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:52.636346 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea2a43d-764c-4c4d-9985-0347ce8f667b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:52.636723 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea2a43d-764c-4c4d-9985-0347ce8f667b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:52.637096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea2a43d-764c-4c4d-9985-0347ce8f667b admin admin] PATH_INFO: `/v3/projects/cf3422e0aa2840b6b15c1b48f86c476c` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:52.638575 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea2a43d-764c-4c4d-9985-0347ce8f667b admin admin] RBAC: Authorizing `identity:delete_project(project_id=cf3422e0aa2840b6b15c1b48f86c476c)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:52.640552 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fea2a43d-764c-4c4d-9985-0347ce8f667b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:52.821930 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fea2a43d-764c-4c4d-9985-0347ce8f667b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cf3422e0aa2840b6b15c1b48f86c476c', 'request_id': 'req-fea2a43d-764c-4c4d-9985-0347ce8f667b'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:52.822142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3f015424-f029-467b-b2a9-5f3ed861328d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6e47f07a139447fabc55bee0cff8909d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:52.915704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fea2a43d-764c-4c4d-9985-0347ce8f667b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cf3422e0aa2840b6b15c1b48f86c476c', 'request_id': 'req-fea2a43d-764c-4c4d-9985-0347ce8f667b'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:52.923184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fea2a43d-764c-4c4d-9985-0347ce8f667b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cf3422e0aa2840b6b15c1b48f86c476c', 'request_id': 'req-fea2a43d-764c-4c4d-9985-0347ce8f667b'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:52.934346 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fea2a43d-764c-4c4d-9985-0347ce8f667b admin admin] The token cache is being invalidate because project cf3422e0aa2840b6b15c1b48f86c476c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:52.934676 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fea2a43d-764c-4c4d-9985-0347ce8f667b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:52.935327 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-fea2a43d-764c-4c4d-9985-0347ce8f667b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:52.936805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1155/2297] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:52 2022] DELETE /identity/v3/projects/cf3422e0aa2840b6b15c1b48f86c476c => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:52.940575 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea2a43d-764c-4c4d-9985-0347ce8f667b admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:52.941845 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-fea2a43d-764c-4c4d-9985-0347ce8f667b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:52.972892 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3f015424-f029-467b-b2a9-5f3ed861328d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6e47f07a139447fabc55bee0cff8909d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:52.973781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3f015424-f029-467b-b2a9-5f3ed861328d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6e47f07a139447fabc55bee0cff8909d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:52.989336 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3f015424-f029-467b-b2a9-5f3ed861328d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6e47f07a139447fabc55bee0cff8909d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:52.996630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3f015424-f029-467b-b2a9-5f3ed861328d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6e47f07a139447fabc55bee0cff8909d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:53.002969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1143/2298] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:52 2022] DELETE /identity/v3/users/6e47f07a139447fabc55bee0cff8909d => generated 0 bytes in 549 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:53.016238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea2a43d-764c-4c4d-9985-0347ce8f667b admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:53.074907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bdb870-4d06-49e7-810d-455e4dfc63b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:53.076029 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bdb870-4d06-49e7-810d-455e4dfc63b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:53.076306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bdb870-4d06-49e7-810d-455e4dfc63b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:53.076576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bdb870-4d06-49e7-810d-455e4dfc63b1 admin admin] PATH_INFO: `/v3/users/2c11737581e443fd81a439148fd4d0f0` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:53.079238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bdb870-4d06-49e7-810d-455e4dfc63b1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c11737581e443fd81a439148fd4d0f0)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:53.082076 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bdb870-4d06-49e7-810d-455e4dfc63b1 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:53.340202 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-71bdb870-4d06-49e7-810d-455e4dfc63b1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c11737581e443fd81a439148fd4d0f0'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:53.433330 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f015424-f029-467b-b2a9-5f3ed861328d admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:53.436658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3f015424-f029-467b-b2a9-5f3ed861328d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:53.452585 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f015424-f029-467b-b2a9-5f3ed861328d admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:53.465508 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2084a23a-ecb3-452d-8745-9fc8a42bf030 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:53.466796 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2084a23a-ecb3-452d-8745-9fc8a42bf030 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:53.472549 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-71bdb870-4d06-49e7-810d-455e4dfc63b1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c11737581e443fd81a439148fd4d0f0'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:53.473163 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2084a23a-ecb3-452d-8745-9fc8a42bf030 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:53.473163 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2084a23a-ecb3-452d-8745-9fc8a42bf030 admin admin] PATH_INFO: `/v3/projects/662feae02a0c4b58afb4ce57482df1d2` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:53.473163 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2084a23a-ecb3-452d-8745-9fc8a42bf030 admin admin] RBAC: Authorizing `identity:delete_project(project_id=662feae02a0c4b58afb4ce57482df1d2)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:53.473163 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2084a23a-ecb3-452d-8745-9fc8a42bf030 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:53.495010 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-71bdb870-4d06-49e7-810d-455e4dfc63b1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c11737581e443fd81a439148fd4d0f0'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:53.495922 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-71bdb870-4d06-49e7-810d-455e4dfc63b1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c11737581e443fd81a439148fd4d0f0'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:53.503472 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-71bdb870-4d06-49e7-810d-455e4dfc63b1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c11737581e443fd81a439148fd4d0f0'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:53.510730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1156/2299] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:52 2022] DELETE /identity/v3/users/2c11737581e443fd81a439148fd4d0f0 => generated 0 bytes in 570 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:53.575742 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2084a23a-ecb3-452d-8745-9fc8a42bf030 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '662feae02a0c4b58afb4ce57482df1d2', 'request_id': 'req-2084a23a-ecb3-452d-8745-9fc8a42bf030'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:53.607202 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:53.623801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2084a23a-ecb3-452d-8745-9fc8a42bf030 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '662feae02a0c4b58afb4ce57482df1d2', 'request_id': 'req-2084a23a-ecb3-452d-8745-9fc8a42bf030'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:53.630395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2084a23a-ecb3-452d-8745-9fc8a42bf030 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '662feae02a0c4b58afb4ce57482df1d2', 'request_id': 'req-2084a23a-ecb3-452d-8745-9fc8a42bf030'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:53.642660 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2084a23a-ecb3-452d-8745-9fc8a42bf030 admin admin] The token cache is being invalidate because project 662feae02a0c4b58afb4ce57482df1d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:53.643059 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2084a23a-ecb3-452d-8745-9fc8a42bf030 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:53.643723 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2084a23a-ecb3-452d-8745-9fc8a42bf030 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:53.645737 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1144/2300] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:53 2022] DELETE /identity/v3/projects/662feae02a0c4b58afb4ce57482df1d2 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:53.649835 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bdb870-4d06-49e7-810d-455e4dfc63b1 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:53.650179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:53.651318 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-71bdb870-4d06-49e7-810d-455e4dfc63b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:53.725752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bdb870-4d06-49e7-810d-455e4dfc63b1 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:53.794042 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f4c2d0-4fc7-45e2-b55f-caedbbc98ed3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:53.795075 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f4c2d0-4fc7-45e2-b55f-caedbbc98ed3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:53.795495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f4c2d0-4fc7-45e2-b55f-caedbbc98ed3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:53.795868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f4c2d0-4fc7-45e2-b55f-caedbbc98ed3 admin admin] PATH_INFO: `/v3/users/b8887c97db054b3fa0ca6bf9c5d5db85` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:53.797634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f4c2d0-4fc7-45e2-b55f-caedbbc98ed3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b8887c97db054b3fa0ca6bf9c5d5db85)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:53.799992 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f4c2d0-4fc7-45e2-b55f-caedbbc98ed3 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:53.966066 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2084a23a-ecb3-452d-8745-9fc8a42bf030 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:53.971271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2084a23a-ecb3-452d-8745-9fc8a42bf030 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:53.983713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2084a23a-ecb3-452d-8745-9fc8a42bf030 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:53.992321 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f360878a-ca39-4a14-a238-5712ec49f790 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:53.992986 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f360878a-ca39-4a14-a238-5712ec49f790 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:53.993270 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f360878a-ca39-4a14-a238-5712ec49f790 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:53.993567 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f360878a-ca39-4a14-a238-5712ec49f790 admin admin] PATH_INFO: `/v3/projects/aa0d5c5b30164752bd874e3568d19891` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:53.995228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f360878a-ca39-4a14-a238-5712ec49f790 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aa0d5c5b30164752bd874e3568d19891)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:54.000317 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f360878a-ca39-4a14-a238-5712ec49f790 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:54.006659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-27f4c2d0-4fc7-45e2-b55f-caedbbc98ed3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b8887c97db054b3fa0ca6bf9c5d5db85'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:54.081591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-27f4c2d0-4fc7-45e2-b55f-caedbbc98ed3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b8887c97db054b3fa0ca6bf9c5d5db85'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:54.098823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-27f4c2d0-4fc7-45e2-b55f-caedbbc98ed3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b8887c97db054b3fa0ca6bf9c5d5db85'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:54.099556 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-27f4c2d0-4fc7-45e2-b55f-caedbbc98ed3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b8887c97db054b3fa0ca6bf9c5d5db85'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:54.106285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-27f4c2d0-4fc7-45e2-b55f-caedbbc98ed3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b8887c97db054b3fa0ca6bf9c5d5db85'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:54.112324 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1157/2301] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:53 2022] DELETE /identity/v3/users/b8887c97db054b3fa0ca6bf9c5d5db85 => generated 0 bytes in 463 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:54.154166 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f360878a-ca39-4a14-a238-5712ec49f790 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aa0d5c5b30164752bd874e3568d19891', 'request_id': 'req-f360878a-ca39-4a14-a238-5712ec49f790'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:54.258863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f360878a-ca39-4a14-a238-5712ec49f790 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aa0d5c5b30164752bd874e3568d19891', 'request_id': 'req-f360878a-ca39-4a14-a238-5712ec49f790'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:54.268830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f360878a-ca39-4a14-a238-5712ec49f790 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aa0d5c5b30164752bd874e3568d19891', 'request_id': 'req-f360878a-ca39-4a14-a238-5712ec49f790'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:54.283168 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f360878a-ca39-4a14-a238-5712ec49f790 admin admin] The token cache is being invalidate because project aa0d5c5b30164752bd874e3568d19891 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:54.283544 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f360878a-ca39-4a14-a238-5712ec49f790 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:54.284121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f360878a-ca39-4a14-a238-5712ec49f790 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:54.285621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1145/2302] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:53 2022] DELETE /identity/v3/projects/aa0d5c5b30164752bd874e3568d19891 => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:54.291645 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de320fba-651f-438d-9ed0-f6320175894e None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:54.291943 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de320fba-651f-438d-9ed0-f6320175894e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:54.292209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de320fba-651f-438d-9ed0-f6320175894e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:54.315752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-de320fba-651f-438d-9ed0-f6320175894e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:54.317141 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-de320fba-651f-438d-9ed0-f6320175894e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:54.357294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:54.397734 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1158/2303] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:03:54.406135 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226146f9-1d6b-4833-9ff3-14bec6220bee None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:54.406394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226146f9-1d6b-4833-9ff3-14bec6220bee None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:54.406674 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226146f9-1d6b-4833-9ff3-14bec6220bee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:54.428990 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-226146f9-1d6b-4833-9ff3-14bec6220bee None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:54.434324 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-226146f9-1d6b-4833-9ff3-14bec6220bee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:54.444759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1146/2304] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:54.449073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de320fba-651f-438d-9ed0-f6320175894e None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:54.459432 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de320fba-651f-438d-9ed0-f6320175894e None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:54.464395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43c8516-a140-4749-bfb9-e21ecbdd886b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:54.465002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43c8516-a140-4749-bfb9-e21ecbdd886b admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:54.465283 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43c8516-a140-4749-bfb9-e21ecbdd886b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:54.465555 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43c8516-a140-4749-bfb9-e21ecbdd886b admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:54.466293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43c8516-a140-4749-bfb9-e21ecbdd886b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:03:54.466711 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43c8516-a140-4749-bfb9-e21ecbdd886b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:54.468312 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43c8516-a140-4749-bfb9-e21ecbdd886b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:54.482064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1159/2305] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:03:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 14:03:54.488986 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226146f9-1d6b-4833-9ff3-14bec6220bee None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:54.499003 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226146f9-1d6b-4833-9ff3-14bec6220bee None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:54.503936 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d66851b-99e7-4489-aa0c-f09c5c00d5c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:54.504757 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d66851b-99e7-4489-aa0c-f09c5c00d5c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:54.505011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d66851b-99e7-4489-aa0c-f09c5c00d5c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:54.505263 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d66851b-99e7-4489-aa0c-f09c5c00d5c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:54.505897 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d66851b-99e7-4489-aa0c-f09c5c00d5c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:54.507810 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d66851b-99e7-4489-aa0c-f09c5c00d5c6 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:54.567572 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43c8516-a140-4749-bfb9-e21ecbdd886b admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:54.571543 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f43c8516-a140-4749-bfb9-e21ecbdd886b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:54.586034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43c8516-a140-4749-bfb9-e21ecbdd886b admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:54.591048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503a84eb-b8bd-4a1c-b579-146dbfe4f0c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:54.591686 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503a84eb-b8bd-4a1c-b579-146dbfe4f0c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:54.593094 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503a84eb-b8bd-4a1c-b579-146dbfe4f0c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:54.593381 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503a84eb-b8bd-4a1c-b579-146dbfe4f0c2 admin admin] PATH_INFO: `/v3/projects/1fc3dc8ec8de4206a4c3a12dbb2a6e0d` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:54.594725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503a84eb-b8bd-4a1c-b579-146dbfe4f0c2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1fc3dc8ec8de4206a4c3a12dbb2a6e0d)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:54.596716 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503a84eb-b8bd-4a1c-b579-146dbfe4f0c2 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:54.618363 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1147/2306] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:03:54 2022] POST /identity/v3/projects => generated 373 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:03:54.622414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d66851b-99e7-4489-aa0c-f09c5c00d5c6 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:54.696508 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-503a84eb-b8bd-4a1c-b579-146dbfe4f0c2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1fc3dc8ec8de4206a4c3a12dbb2a6e0d', 'request_id': 'req-503a84eb-b8bd-4a1c-b579-146dbfe4f0c2'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:54.696766 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d66851b-99e7-4489-aa0c-f09c5c00d5c6 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:54.722687 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:54.757931 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933abcc8-17e1-46b2-8025-151f44af7979 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:54.758584 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933abcc8-17e1-46b2-8025-151f44af7979 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:54.759061 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933abcc8-17e1-46b2-8025-151f44af7979 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:54.759337 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933abcc8-17e1-46b2-8025-151f44af7979 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:54.760041 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933abcc8-17e1-46b2-8025-151f44af7979 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:54.762451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933abcc8-17e1-46b2-8025-151f44af7979 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:54.771698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-503a84eb-b8bd-4a1c-b579-146dbfe4f0c2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1fc3dc8ec8de4206a4c3a12dbb2a6e0d', 'request_id': 'req-503a84eb-b8bd-4a1c-b579-146dbfe4f0c2'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:54.777820 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-503a84eb-b8bd-4a1c-b579-146dbfe4f0c2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1fc3dc8ec8de4206a4c3a12dbb2a6e0d', 'request_id': 'req-503a84eb-b8bd-4a1c-b579-146dbfe4f0c2'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:54.787193 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:54.791768 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-503a84eb-b8bd-4a1c-b579-146dbfe4f0c2 admin admin] The token cache is being invalidate because project 1fc3dc8ec8de4206a4c3a12dbb2a6e0d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:54.792055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-503a84eb-b8bd-4a1c-b579-146dbfe4f0c2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:54.792719 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-503a84eb-b8bd-4a1c-b579-146dbfe4f0c2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:54.794919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1160/2307] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:54 2022] DELETE /identity/v3/projects/1fc3dc8ec8de4206a4c3a12dbb2a6e0d => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:03:54.871030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1148/2308] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:03:54 2022] POST /identity/v3/users => generated 287 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:03:54.876371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503a84eb-b8bd-4a1c-b579-146dbfe4f0c2 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:54.877819 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-503a84eb-b8bd-4a1c-b579-146dbfe4f0c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:54.891870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503a84eb-b8bd-4a1c-b579-146dbfe4f0c2 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:54.897804 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534d8240-2fb1-480d-84e0-d878b966f0f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:54.898486 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534d8240-2fb1-480d-84e0-d878b966f0f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:54.898771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534d8240-2fb1-480d-84e0-d878b966f0f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:54.899043 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534d8240-2fb1-480d-84e0-d878b966f0f8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:54.899891 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534d8240-2fb1-480d-84e0-d878b966f0f8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:54.901579 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534d8240-2fb1-480d-84e0-d878b966f0f8 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:54.913558 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1161/2309] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:54 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:54.917862 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933abcc8-17e1-46b2-8025-151f44af7979 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:54.934405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933abcc8-17e1-46b2-8025-151f44af7979 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:54.939673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c227fc2-e4f6-4231-a7d0-6c6289988550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:54.940422 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c227fc2-e4f6-4231-a7d0-6c6289988550 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:54.940781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c227fc2-e4f6-4231-a7d0-6c6289988550 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:54.941115 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c227fc2-e4f6-4231-a7d0-6c6289988550 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:54.942226 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c227fc2-e4f6-4231-a7d0-6c6289988550 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:54.943888 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c227fc2-e4f6-4231-a7d0-6c6289988550 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:54.955683 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1149/2310] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:03:54 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:03:54.963861 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534d8240-2fb1-480d-84e0-d878b966f0f8 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:54.975354 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534d8240-2fb1-480d-84e0-d878b966f0f8 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:55.001274 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf3479e-bd6a-428f-84e9-040334a3396d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:55.002264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf3479e-bd6a-428f-84e9-040334a3396d admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:55.002580 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf3479e-bd6a-428f-84e9-040334a3396d admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:55.002881 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf3479e-bd6a-428f-84e9-040334a3396d admin admin] PATH_INFO: `/v3/projects/2373ab1200e74839b2ea1b51162a18e4/users/1b602e34b1b64efe8049ba0c980d8b31/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:55.021506 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf3479e-bd6a-428f-84e9-040334a3396d admin admin] RBAC: Authorizing `identity:create_grant(project_id=2373ab1200e74839b2ea1b51162a18e4, user_id=1b602e34b1b64efe8049ba0c980d8b31, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:55.024423 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf3479e-bd6a-428f-84e9-040334a3396d admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:55.099300 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1162/2311] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:03:54 2022] PUT /identity/v3/projects/2373ab1200e74839b2ea1b51162a18e4/users/1b602e34b1b64efe8049ba0c980d8b31/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:55.109346 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4e9ab3-4244-4f04-8bff-c79cff3eb554 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:55.109840 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4e9ab3-4244-4f04-8bff-c79cff3eb554 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:55.109936 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4e9ab3-4244-4f04-8bff-c79cff3eb554 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:55.145511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-8f4e9ab3-4244-4f04-8bff-c79cff3eb554 None None] MFA Rules not processed for user `1b602e34b1b64efe8049ba0c980d8b31`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:55.146754 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8f4e9ab3-4244-4f04-8bff-c79cff3eb554 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:55.212804 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1150/2312] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:55 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:55.224359 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf3479e-bd6a-428f-84e9-040334a3396d admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:55.226039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-6bf3479e-bd6a-428f-84e9-040334a3396d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:55.288258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:55.292036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf3479e-bd6a-428f-84e9-040334a3396d admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:55.296131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e21eea-825c-4319-9ac4-280ca0fec8fd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:55.298859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e21eea-825c-4319-9ac4-280ca0fec8fd service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:55.298859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e21eea-825c-4319-9ac4-280ca0fec8fd service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:55.298859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e21eea-825c-4319-9ac4-280ca0fec8fd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:55.305825 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e21eea-825c-4319-9ac4-280ca0fec8fd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:55.307275 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e21eea-825c-4319-9ac4-280ca0fec8fd service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:55.316286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1163/2313] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:55 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:55.345038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4e9ab3-4244-4f04-8bff-c79cff3eb554 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:55.372934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:DELETE=2,keystone:SELECT=95,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:55.408706 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4e9ab3-4244-4f04-8bff-c79cff3eb554 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:55.413986 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1af396-8432-4da5-bc5c-4b5e8c3f9331 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:55.414650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1af396-8432-4da5-bc5c-4b5e8c3f9331 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:55.414929 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1af396-8432-4da5-bc5c-4b5e8c3f9331 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:55.415196 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1af396-8432-4da5-bc5c-4b5e8c3f9331 admin admin] PATH_INFO: `/v3/users/1b602e34b1b64efe8049ba0c980d8b31` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:55.416333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1af396-8432-4da5-bc5c-4b5e8c3f9331 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1b602e34b1b64efe8049ba0c980d8b31)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:55.418238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1af396-8432-4da5-bc5c-4b5e8c3f9331 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:55.458570 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:55.580332 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:55.584298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:55.589457 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-ed1af396-8432-4da5-bc5c-4b5e8c3f9331 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1b602e34b1b64efe8049ba0c980d8b31'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:55.652481 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-ed1af396-8432-4da5-bc5c-4b5e8c3f9331 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1b602e34b1b64efe8049ba0c980d8b31'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:55.653212 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-ed1af396-8432-4da5-bc5c-4b5e8c3f9331 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1b602e34b1b64efe8049ba0c980d8b31'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:55.664331 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:55.666131 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-ed1af396-8432-4da5-bc5c-4b5e8c3f9331 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1b602e34b1b64efe8049ba0c980d8b31'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:55.673541 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-ed1af396-8432-4da5-bc5c-4b5e8c3f9331 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1b602e34b1b64efe8049ba0c980d8b31'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:55.679747 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1151/2314] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:03:55 2022] DELETE /identity/v3/users/1b602e34b1b64efe8049ba0c980d8b31 => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:03:55.685568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20575d24-869f-4d2a-9f71-ef6cdd0a74a4 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:55.685892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20575d24-869f-4d2a-9f71-ef6cdd0a74a4 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:55.686124 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20575d24-869f-4d2a-9f71-ef6cdd0a74a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:55.705849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-20575d24-869f-4d2a-9f71-ef6cdd0a74a4 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:03:55.707098 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-20575d24-869f-4d2a-9f71-ef6cdd0a74a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:55.767200 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1164/2315] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:03:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:03:55.777741 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1af396-8432-4da5-bc5c-4b5e8c3f9331 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:55.780440 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ed1af396-8432-4da5-bc5c-4b5e8c3f9331 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:55.824208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:55.842225 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1af396-8432-4da5-bc5c-4b5e8c3f9331 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:55.845904 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74009c27-c6f6-4463-90ae-e91db6735fa3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:55.846538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74009c27-c6f6-4463-90ae-e91db6735fa3 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:55.846816 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74009c27-c6f6-4463-90ae-e91db6735fa3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:55.847118 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74009c27-c6f6-4463-90ae-e91db6735fa3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:55.856710 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74009c27-c6f6-4463-90ae-e91db6735fa3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:55.858144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74009c27-c6f6-4463-90ae-e91db6735fa3 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:55.867602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1152/2316] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:03:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:03:55.910098 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:56.347403 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20575d24-869f-4d2a-9f71-ef6cdd0a74a4 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:03:56.348845 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-20575d24-869f-4d2a-9f71-ef6cdd0a74a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:03:56.366562 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20575d24-869f-4d2a-9f71-ef6cdd0a74a4 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:03:56.371329 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ef08a4-0e1a-4e45-9944-4be4a25107dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:03:56.371954 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ef08a4-0e1a-4e45-9944-4be4a25107dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:03:56.372235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ef08a4-0e1a-4e45-9944-4be4a25107dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:03:56.372773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ef08a4-0e1a-4e45-9944-4be4a25107dc admin admin] PATH_INFO: `/v3/projects/2373ab1200e74839b2ea1b51162a18e4` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:03:56.373622 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50ef08a4-0e1a-4e45-9944-4be4a25107dc admin admin] RBAC: Authorizing `identity:delete_project(project_id=2373ab1200e74839b2ea1b51162a18e4)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:03:56.375585 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50ef08a4-0e1a-4e45-9944-4be4a25107dc admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:03:56.394408 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:DELETE=6,keystone:SELECT=92,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:03:56.519078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-50ef08a4-0e1a-4e45-9944-4be4a25107dc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2373ab1200e74839b2ea1b51162a18e4', 'request_id': 'req-50ef08a4-0e1a-4e45-9944-4be4a25107dc'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:56.583537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-50ef08a4-0e1a-4e45-9944-4be4a25107dc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2373ab1200e74839b2ea1b51162a18e4', 'request_id': 'req-50ef08a4-0e1a-4e45-9944-4be4a25107dc'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:56.591306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-50ef08a4-0e1a-4e45-9944-4be4a25107dc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2373ab1200e74839b2ea1b51162a18e4', 'request_id': 'req-50ef08a4-0e1a-4e45-9944-4be4a25107dc'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:56.612565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-50ef08a4-0e1a-4e45-9944-4be4a25107dc admin admin] The token cache is being invalidate because project 2373ab1200e74839b2ea1b51162a18e4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:03:56.612565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-50ef08a4-0e1a-4e45-9944-4be4a25107dc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:56.612565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-50ef08a4-0e1a-4e45-9944-4be4a25107dc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:03:56.612565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1165/2317] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:03:56 2022] DELETE /identity/v3/projects/2373ab1200e74839b2ea1b51162a18e4 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:04:02.706013 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214dce45-8b3a-481e-9c51-28d21ca51f32 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:02.706515 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214dce45-8b3a-481e-9c51-28d21ca51f32 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:02.706603 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214dce45-8b3a-481e-9c51-28d21ca51f32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:02.726305 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-214dce45-8b3a-481e-9c51-28d21ca51f32 None None] MFA Rules not processed for user `0db30f990aae49fabf9b8b19cd019de9`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:04:02.727522 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-214dce45-8b3a-481e-9c51-28d21ca51f32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:02.782316 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1153/2318] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:04:02 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:04:05.251945 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07cc825-c763-46ab-8630-14ae70ddb0dc None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:05.252493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07cc825-c763-46ab-8630-14ae70ddb0dc None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:05.252493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07cc825-c763-46ab-8630-14ae70ddb0dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:05.271294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-d07cc825-c763-46ab-8630-14ae70ddb0dc None None] MFA Rules not processed for user `9359e90c4861470d9cc060abb9c3b203`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:04:05.272474 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d07cc825-c763-46ab-8630-14ae70ddb0dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:05.329515 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1166/2319] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:04:05 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:04:12.774903 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=66,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:04:15.322190 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=42,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:04:17.316163 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3d7815-6c7f-4b30-b7af-6fc2c6a725a0 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:17.316688 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3d7815-6c7f-4b30-b7af-6fc2c6a725a0 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:17.316688 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3d7815-6c7f-4b30-b7af-6fc2c6a725a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:17.340912 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-2b3d7815-6c7f-4b30-b7af-6fc2c6a725a0 None None] MFA Rules not processed for user `28b64a35d7b44519a1d93919f2e23083`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:04:17.342156 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2b3d7815-6c7f-4b30-b7af-6fc2c6a725a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:17.406320 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1154/2320] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:04:17 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:04:24.654533 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07cc825-c763-46ab-8630-14ae70ddb0dc None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:24.656026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d07cc825-c763-46ab-8630-14ae70ddb0dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:24.726012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07cc825-c763-46ab-8630-14ae70ddb0dc None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:24.736176 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95cebe4-84a2-423d-a3c4-f9155cfb0591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:24.736606 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95cebe4-84a2-423d-a3c4-f9155cfb0591 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:24.736945 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95cebe4-84a2-423d-a3c4-f9155cfb0591 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:24.737147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95cebe4-84a2-423d-a3c4-f9155cfb0591 admin admin] PATH_INFO: `/v3/users/843c85a2734e405a8649c9e5dcd2c183` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:24.738262 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95cebe4-84a2-423d-a3c4-f9155cfb0591 admin admin] RBAC: Authorizing `identity:delete_user(user_id=843c85a2734e405a8649c9e5dcd2c183)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:24.740200 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95cebe4-84a2-423d-a3c4-f9155cfb0591 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:24.989256 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-d95cebe4-84a2-423d-a3c4-f9155cfb0591 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '843c85a2734e405a8649c9e5dcd2c183'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:04:25.078587 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-d95cebe4-84a2-423d-a3c4-f9155cfb0591 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '843c85a2734e405a8649c9e5dcd2c183'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:04:25.092971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-d95cebe4-84a2-423d-a3c4-f9155cfb0591 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '843c85a2734e405a8649c9e5dcd2c183'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:04:25.096895 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-d95cebe4-84a2-423d-a3c4-f9155cfb0591 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '843c85a2734e405a8649c9e5dcd2c183'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:04:25.105218 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-d95cebe4-84a2-423d-a3c4-f9155cfb0591 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '843c85a2734e405a8649c9e5dcd2c183'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:04:25.111682 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1167/2321] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:04:24 2022] DELETE /identity/v3/users/843c85a2734e405a8649c9e5dcd2c183 => generated 0 bytes in 458 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:04:25.116224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e6ba68-0725-48c5-bd5e-f12bb5189b98 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:25.117160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e6ba68-0725-48c5-bd5e-f12bb5189b98 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:25.117160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e6ba68-0725-48c5-bd5e-f12bb5189b98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:25.137875 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-98e6ba68-0725-48c5-bd5e-f12bb5189b98 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:04:25.139174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-98e6ba68-0725-48c5-bd5e-f12bb5189b98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:25.202618 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1155/2322] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:04:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:04:25.212388 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95cebe4-84a2-423d-a3c4-f9155cfb0591 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:25.213792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d95cebe4-84a2-423d-a3c4-f9155cfb0591 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:25.275153 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95cebe4-84a2-423d-a3c4-f9155cfb0591 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:25.278855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c2d41da-e4f2-44ea-aab7-3d516242ecd1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:25.279456 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2d41da-e4f2-44ea-aab7-3d516242ecd1 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:25.279730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2d41da-e4f2-44ea-aab7-3d516242ecd1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:25.279995 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c2d41da-e4f2-44ea-aab7-3d516242ecd1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:25.289616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2d41da-e4f2-44ea-aab7-3d516242ecd1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:25.291151 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c2d41da-e4f2-44ea-aab7-3d516242ecd1 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:25.302172 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1168/2323] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:04:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:04:25.749841 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e6ba68-0725-48c5-bd5e-f12bb5189b98 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:25.751332 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-98e6ba68-0725-48c5-bd5e-f12bb5189b98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:25.768929 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e6ba68-0725-48c5-bd5e-f12bb5189b98 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:25.774089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a6280d-0e1b-4800-a2d7-8747c7a68082 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:25.774764 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a6280d-0e1b-4800-a2d7-8747c7a68082 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:25.775041 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a6280d-0e1b-4800-a2d7-8747c7a68082 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:25.775309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a6280d-0e1b-4800-a2d7-8747c7a68082 admin admin] PATH_INFO: `/v3/projects/f045ee2237084f32a053b50c8299abff` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:25.776575 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a6280d-0e1b-4800-a2d7-8747c7a68082 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f045ee2237084f32a053b50c8299abff)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:25.778710 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a6280d-0e1b-4800-a2d7-8747c7a68082 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:25.929622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-00a6280d-0e1b-4800-a2d7-8747c7a68082 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f045ee2237084f32a053b50c8299abff', 'request_id': 'req-00a6280d-0e1b-4800-a2d7-8747c7a68082'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:04:26.002892 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-00a6280d-0e1b-4800-a2d7-8747c7a68082 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f045ee2237084f32a053b50c8299abff', 'request_id': 'req-00a6280d-0e1b-4800-a2d7-8747c7a68082'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:04:26.008983 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-00a6280d-0e1b-4800-a2d7-8747c7a68082 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f045ee2237084f32a053b50c8299abff', 'request_id': 'req-00a6280d-0e1b-4800-a2d7-8747c7a68082'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:04:26.025685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-00a6280d-0e1b-4800-a2d7-8747c7a68082 admin admin] The token cache is being invalidate because project f045ee2237084f32a053b50c8299abff was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:04:26.025972 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-00a6280d-0e1b-4800-a2d7-8747c7a68082 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:04:26.027084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-00a6280d-0e1b-4800-a2d7-8747c7a68082 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:04:26.028333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1156/2324] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:04:25 2022] DELETE /identity/v3/projects/f045ee2237084f32a053b50c8299abff => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:04:26.033284 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac0439a-93d4-4c5a-813c-6bf231ba6e12 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:26.033767 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac0439a-93d4-4c5a-813c-6bf231ba6e12 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:26.033861 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac0439a-93d4-4c5a-813c-6bf231ba6e12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:26.060442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-8ac0439a-93d4-4c5a-813c-6bf231ba6e12 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:04:26.061686 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8ac0439a-93d4-4c5a-813c-6bf231ba6e12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:26.079715 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:04:26.125514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1169/2325] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:04:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:04:26.133481 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6f53a6-6c6e-4df9-bd0c-493b8ec343f7 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:26.133964 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6f53a6-6c6e-4df9-bd0c-493b8ec343f7 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:26.134079 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6f53a6-6c6e-4df9-bd0c-493b8ec343f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:26.156044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-3e6f53a6-6c6e-4df9-bd0c-493b8ec343f7 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:04:26.157386 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3e6f53a6-6c6e-4df9-bd0c-493b8ec343f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:26.166648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1157/2326] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:04:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:04:26.172244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac0439a-93d4-4c5a-813c-6bf231ba6e12 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:26.182068 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac0439a-93d4-4c5a-813c-6bf231ba6e12 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:26.187272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1dbab7d-1fac-4052-a2f7-a931cbd141be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:26.187860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dbab7d-1fac-4052-a2f7-a931cbd141be admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:26.188136 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dbab7d-1fac-4052-a2f7-a931cbd141be admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:26.188403 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1dbab7d-1fac-4052-a2f7-a931cbd141be admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:26.189036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1dbab7d-1fac-4052-a2f7-a931cbd141be admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:04:26.189333 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1dbab7d-1fac-4052-a2f7-a931cbd141be admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:26.190964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1dbab7d-1fac-4052-a2f7-a931cbd141be admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:26.203002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1170/2327] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:04:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:04:26.207436 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6f53a6-6c6e-4df9-bd0c-493b8ec343f7 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:26.219170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6f53a6-6c6e-4df9-bd0c-493b8ec343f7 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:26.224105 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80aa51af-737b-4634-b790-d3a31c612eea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:26.224705 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aa51af-737b-4634-b790-d3a31c612eea admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:26.225204 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aa51af-737b-4634-b790-d3a31c612eea admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:26.225478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80aa51af-737b-4634-b790-d3a31c612eea admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:26.226286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80aa51af-737b-4634-b790-d3a31c612eea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:26.228767 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80aa51af-737b-4634-b790-d3a31c612eea admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:26.306932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:04:26.313298 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1158/2328] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:04:26 2022] POST /identity/v3/projects => generated 373 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:04:26.317578 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1dbab7d-1fac-4052-a2f7-a931cbd141be admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:26.327991 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1dbab7d-1fac-4052-a2f7-a931cbd141be admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:26.388113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5308142f-cb22-4f6a-8fe9-5eb7ee78821a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:26.388735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5308142f-cb22-4f6a-8fe9-5eb7ee78821a admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:26.388990 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5308142f-cb22-4f6a-8fe9-5eb7ee78821a admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:26.389256 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5308142f-cb22-4f6a-8fe9-5eb7ee78821a admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:26.390019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5308142f-cb22-4f6a-8fe9-5eb7ee78821a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:26.391966 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5308142f-cb22-4f6a-8fe9-5eb7ee78821a admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:26.507205 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1171/2329] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:04:26 2022] POST /identity/v3/users => generated 287 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:04:26.512127 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80aa51af-737b-4634-b790-d3a31c612eea admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:26.522383 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80aa51af-737b-4634-b790-d3a31c612eea admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:26.527644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bbada5-51fa-4049-94d0-18ea91173c35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:26.528239 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bbada5-51fa-4049-94d0-18ea91173c35 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:26.528516 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bbada5-51fa-4049-94d0-18ea91173c35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:26.528787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bbada5-51fa-4049-94d0-18ea91173c35 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:26.529425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bbada5-51fa-4049-94d0-18ea91173c35 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:26.531115 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4bbada5-51fa-4049-94d0-18ea91173c35 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:26.541074 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1159/2330] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:04:26 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:04:26.545690 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5308142f-cb22-4f6a-8fe9-5eb7ee78821a admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:26.557394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5308142f-cb22-4f6a-8fe9-5eb7ee78821a admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:26.562519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735eb665-1683-4552-8384-5601c7a9025a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:26.563282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735eb665-1683-4552-8384-5601c7a9025a admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:26.563553 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735eb665-1683-4552-8384-5601c7a9025a admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:26.563841 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735eb665-1683-4552-8384-5601c7a9025a admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:26.564461 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735eb665-1683-4552-8384-5601c7a9025a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:26.566741 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735eb665-1683-4552-8384-5601c7a9025a admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:26.581475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1172/2331] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:04:26 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:04:26.582658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bbada5-51fa-4049-94d0-18ea91173c35 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:26.594193 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bbada5-51fa-4049-94d0-18ea91173c35 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:26.599387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ef31f7-7372-461d-a22e-6a2c7e2daa7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:26.600043 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ef31f7-7372-461d-a22e-6a2c7e2daa7a admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:26.600326 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ef31f7-7372-461d-a22e-6a2c7e2daa7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:26.600601 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ef31f7-7372-461d-a22e-6a2c7e2daa7a admin admin] PATH_INFO: `/v3/projects/8f795f5f1a0541e28e28c4fc48877888/users/e4de1df05f09452fa54a560f1934b62d/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:26.621365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ef31f7-7372-461d-a22e-6a2c7e2daa7a admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f795f5f1a0541e28e28c4fc48877888, user_id=e4de1df05f09452fa54a560f1934b62d, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:26.624082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ef31f7-7372-461d-a22e-6a2c7e2daa7a admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:26.725462 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1160/2332] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:04:26 2022] PUT /identity/v3/projects/8f795f5f1a0541e28e28c4fc48877888/users/e4de1df05f09452fa54a560f1934b62d/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:04:26.735966 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f224f1-a138-4dc6-ac25-1f05da80d970 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:26.736253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f224f1-a138-4dc6-ac25-1f05da80d970 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:26.736530 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f224f1-a138-4dc6-ac25-1f05da80d970 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:26.773497 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-b1f224f1-a138-4dc6-ac25-1f05da80d970 None None] MFA Rules not processed for user `e4de1df05f09452fa54a560f1934b62d`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:04:26.774836 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-b1f224f1-a138-4dc6-ac25-1f05da80d970 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:26.842147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1173/2333] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:04:26 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:04:26.852994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ef31f7-7372-461d-a22e-6a2c7e2daa7a admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:26.855280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-51ef31f7-7372-461d-a22e-6a2c7e2daa7a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:26.919604 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ef31f7-7372-461d-a22e-6a2c7e2daa7a admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:26.923215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b78d762-96de-40ec-a3b5-f5219e4774a5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:26.923812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b78d762-96de-40ec-a3b5-f5219e4774a5 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:26.924087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b78d762-96de-40ec-a3b5-f5219e4774a5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:26.924355 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b78d762-96de-40ec-a3b5-f5219e4774a5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:26.932964 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b78d762-96de-40ec-a3b5-f5219e4774a5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:26.934626 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b78d762-96de-40ec-a3b5-f5219e4774a5 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:26.943380 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1161/2334] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:04:26 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:04:28.675458 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7be0fef-b03e-4985-9483-13b55c0aff6a None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:28.675978 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7be0fef-b03e-4985-9483-13b55c0aff6a None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:28.676157 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7be0fef-b03e-4985-9483-13b55c0aff6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:28.695495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-e7be0fef-b03e-4985-9483-13b55c0aff6a None None] MFA Rules not processed for user `e4de1df05f09452fa54a560f1934b62d`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:04:28.696674 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-e7be0fef-b03e-4985-9483-13b55c0aff6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:28.704240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1174/2335] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:04:28 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:04:33.429255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b78d762-96de-40ec-a3b5-f5219e4774a5 service cinder] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:33.430765 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-6b78d762-96de-40ec-a3b5-f5219e4774a5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:33.486794 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b78d762-96de-40ec-a3b5-f5219e4774a5 service cinder] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:33.490515 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc69271e-1de1-4a46-a685-9606d6257cca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:33.491104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc69271e-1de1-4a46-a685-9606d6257cca service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:33.491370 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc69271e-1de1-4a46-a685-9606d6257cca service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:33.491644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc69271e-1de1-4a46-a685-9606d6257cca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:33.493322 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-fc69271e-1de1-4a46-a685-9606d6257cca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:33.555091 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc69271e-1de1-4a46-a685-9606d6257cca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:33.556534 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc69271e-1de1-4a46-a685-9606d6257cca service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:33.566037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1162/2336] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:04:33 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:04:38.690073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=92,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:04:43.561802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:04:50.702449 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f238add1-5405-4d4d-8080-7e5ddd2945dd None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:50.702957 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f238add1-5405-4d4d-8080-7e5ddd2945dd None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:50.702957 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f238add1-5405-4d4d-8080-7e5ddd2945dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:50.722533 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-f238add1-5405-4d4d-8080-7e5ddd2945dd None None] MFA Rules not processed for user `b471d768cd814ff8b4f6bd35d21e24a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:04:50.723738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f238add1-5405-4d4d-8080-7e5ddd2945dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:50.778125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1175/2337] 10.222.0.62 () {58 vars in 1009 bytes} [Fri Sep 23 14:04:50 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:04:50.820789 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc69271e-1de1-4a46-a685-9606d6257cca service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:50.822622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-fc69271e-1de1-4a46-a685-9606d6257cca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:50.892284 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc69271e-1de1-4a46-a685-9606d6257cca service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:50.897256 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1426674-7627-4e01-af18-f27e34b8097d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b9269509f1d454eab7a3008c2ba0039', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:50.897858 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1426674-7627-4e01-af18-f27e34b8097d service swift] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:50.898140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1426674-7627-4e01-af18-f27e34b8097d service swift] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:50.898441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1426674-7627-4e01-af18-f27e34b8097d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:50.906859 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1426674-7627-4e01-af18-f27e34b8097d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:50.908425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1426674-7627-4e01-af18-f27e34b8097d service swift] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:50.916732 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1163/2338] 10.222.0.62 () {60 vars in 1384 bytes} [Fri Sep 23 14:04:50 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 23 14:04:51.796080 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d43eea-5803-469a-ba54-d00a4e6f1101 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:51.796526 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d43eea-5803-469a-ba54-d00a4e6f1101 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:51.796671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d43eea-5803-469a-ba54-d00a4e6f1101 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:51.815854 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-67d43eea-5803-469a-ba54-d00a4e6f1101 None None] MFA Rules not processed for user `b471d768cd814ff8b4f6bd35d21e24a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:04:51.817026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-67d43eea-5803-469a-ba54-d00a4e6f1101 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:51.824652 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1176/2339] 10.222.0.62 () {58 vars in 967 bytes} [Fri Sep 23 14:04:51 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:04:51.837268 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1426674-7627-4e01-af18-f27e34b8097d service swift] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:51.848272 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1426674-7627-4e01-af18-f27e34b8097d service swift] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:51.853494 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74d3175-653a-42af-8ca4-ed2c7ea5abcb service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b9269509f1d454eab7a3008c2ba0039', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:51.854088 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74d3175-653a-42af-8ca4-ed2c7ea5abcb service swift] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:51.854420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74d3175-653a-42af-8ca4-ed2c7ea5abcb service swift] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:51.854651 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74d3175-653a-42af-8ca4-ed2c7ea5abcb service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:51.862569 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74d3175-653a-42af-8ca4-ed2c7ea5abcb service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:51.864132 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74d3175-653a-42af-8ca4-ed2c7ea5abcb service swift] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:51.872551 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1164/2340] 10.222.0.62 () {60 vars in 1384 bytes} [Fri Sep 23 14:04:51 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 23 14:04:52.482197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc4c3fc-6227-4609-88bf-8e010fab34b6 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:52.482513 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc4c3fc-6227-4609-88bf-8e010fab34b6 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:52.482787 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc4c3fc-6227-4609-88bf-8e010fab34b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:52.501824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-5dc4c3fc-6227-4609-88bf-8e010fab34b6 None None] MFA Rules not processed for user `b471d768cd814ff8b4f6bd35d21e24a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:04:52.503025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-5dc4c3fc-6227-4609-88bf-8e010fab34b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:52.509942 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1177/2341] 10.222.0.62 () {58 vars in 1009 bytes} [Fri Sep 23 14:04:52 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:04:52.522515 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74d3175-653a-42af-8ca4-ed2c7ea5abcb service swift] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:52.533598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74d3175-653a-42af-8ca4-ed2c7ea5abcb service swift] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:52.538666 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23eabc0-d393-42d1-a87e-4d0964f8a5ed service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b9269509f1d454eab7a3008c2ba0039', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:52.539257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23eabc0-d393-42d1-a87e-4d0964f8a5ed service swift] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:52.539536 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23eabc0-d393-42d1-a87e-4d0964f8a5ed service swift] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:52.539807 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23eabc0-d393-42d1-a87e-4d0964f8a5ed service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:52.548617 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23eabc0-d393-42d1-a87e-4d0964f8a5ed service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:52.550174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23eabc0-d393-42d1-a87e-4d0964f8a5ed service swift] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:52.558099 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1165/2342] 10.222.0.62 () {60 vars in 1384 bytes} [Fri Sep 23 14:04:52 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 23 14:04:53.220905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e8fe5e-d792-4dfe-84d3-bc844b92b0c1 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:53.227400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e8fe5e-d792-4dfe-84d3-bc844b92b0c1 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:53.227400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e8fe5e-d792-4dfe-84d3-bc844b92b0c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:53.240848 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-64e8fe5e-d792-4dfe-84d3-bc844b92b0c1 None None] MFA Rules not processed for user `b471d768cd814ff8b4f6bd35d21e24a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:04:53.242098 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-64e8fe5e-d792-4dfe-84d3-bc844b92b0c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:53.250140 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1178/2343] 10.222.0.62 () {58 vars in 967 bytes} [Fri Sep 23 14:04:53 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:04:53.263486 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23eabc0-d393-42d1-a87e-4d0964f8a5ed service swift] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:53.273828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23eabc0-d393-42d1-a87e-4d0964f8a5ed service swift] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:53.278880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caaeff02-b37c-4bc4-ac92-a75be384ec6d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b9269509f1d454eab7a3008c2ba0039', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:53.279468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caaeff02-b37c-4bc4-ac92-a75be384ec6d service swift] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:53.279746 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caaeff02-b37c-4bc4-ac92-a75be384ec6d service swift] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:53.280018 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caaeff02-b37c-4bc4-ac92-a75be384ec6d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:53.287854 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caaeff02-b37c-4bc4-ac92-a75be384ec6d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:53.289412 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caaeff02-b37c-4bc4-ac92-a75be384ec6d service swift] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:53.297508 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1166/2344] 10.222.0.62 () {60 vars in 1384 bytes} [Fri Sep 23 14:04:53 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 23 14:04:55.987014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e8fe5e-d792-4dfe-84d3-bc844b92b0c1 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:56.049465 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e8fe5e-d792-4dfe-84d3-bc844b92b0c1 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:56.053847 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cbe87f-bef5-4b75-a47b-e64207b53352 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:56.054497 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cbe87f-bef5-4b75-a47b-e64207b53352 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:56.054799 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cbe87f-bef5-4b75-a47b-e64207b53352 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:56.055067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cbe87f-bef5-4b75-a47b-e64207b53352 admin admin] PATH_INFO: `/v3/users/e4de1df05f09452fa54a560f1934b62d` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:56.056167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cbe87f-bef5-4b75-a47b-e64207b53352 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4de1df05f09452fa54a560f1934b62d)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:56.058073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cbe87f-bef5-4b75-a47b-e64207b53352 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:56.243922 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f6cbe87f-bef5-4b75-a47b-e64207b53352 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4de1df05f09452fa54a560f1934b62d'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:04:56.314618 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f6cbe87f-bef5-4b75-a47b-e64207b53352 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4de1df05f09452fa54a560f1934b62d'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:04:56.327488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f6cbe87f-bef5-4b75-a47b-e64207b53352 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4de1df05f09452fa54a560f1934b62d'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:04:56.328166 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f6cbe87f-bef5-4b75-a47b-e64207b53352 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4de1df05f09452fa54a560f1934b62d'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:04:56.335704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f6cbe87f-bef5-4b75-a47b-e64207b53352 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4de1df05f09452fa54a560f1934b62d'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:04:56.341528 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1179/2345] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:04:55 2022] DELETE /identity/v3/users/e4de1df05f09452fa54a560f1934b62d => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:04:56.346639 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b36411-349f-4064-be98-26c62c399e2e None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:56.347140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b36411-349f-4064-be98-26c62c399e2e None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:56.347234 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b36411-349f-4064-be98-26c62c399e2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:56.366006 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-71b36411-349f-4064-be98-26c62c399e2e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:04:56.367271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-71b36411-349f-4064-be98-26c62c399e2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:56.426027 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1167/2346] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:04:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:04:56.435746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cbe87f-bef5-4b75-a47b-e64207b53352 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:56.437167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f6cbe87f-bef5-4b75-a47b-e64207b53352 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:56.443921 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:04:56.494642 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cbe87f-bef5-4b75-a47b-e64207b53352 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:56.498723 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70e895b-6898-4a24-868a-365aed4893de service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:56.499323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70e895b-6898-4a24-868a-365aed4893de service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:56.499603 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70e895b-6898-4a24-868a-365aed4893de service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:56.499875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70e895b-6898-4a24-868a-365aed4893de service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:56.508800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70e895b-6898-4a24-868a-365aed4893de service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:56.511875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70e895b-6898-4a24-868a-365aed4893de service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:56.520211 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1180/2347] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:04:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:04:56.779407 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b36411-349f-4064-be98-26c62c399e2e None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:56.782590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-71b36411-349f-4064-be98-26c62c399e2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:56.793425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b36411-349f-4064-be98-26c62c399e2e None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:56.798470 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04f3d59-3ea8-4963-aa8d-c50c68766da3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:56.799121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04f3d59-3ea8-4963-aa8d-c50c68766da3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:56.799373 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04f3d59-3ea8-4963-aa8d-c50c68766da3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:56.799642 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04f3d59-3ea8-4963-aa8d-c50c68766da3 admin admin] PATH_INFO: `/v3/projects/8f795f5f1a0541e28e28c4fc48877888` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:56.800759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04f3d59-3ea8-4963-aa8d-c50c68766da3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8f795f5f1a0541e28e28c4fc48877888)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:56.802685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04f3d59-3ea8-4963-aa8d-c50c68766da3 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:56.945068 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:04:56.947947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e04f3d59-3ea8-4963-aa8d-c50c68766da3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8f795f5f1a0541e28e28c4fc48877888', 'request_id': 'req-e04f3d59-3ea8-4963-aa8d-c50c68766da3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:04:57.042406 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e04f3d59-3ea8-4963-aa8d-c50c68766da3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8f795f5f1a0541e28e28c4fc48877888', 'request_id': 'req-e04f3d59-3ea8-4963-aa8d-c50c68766da3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:04:57.048706 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e04f3d59-3ea8-4963-aa8d-c50c68766da3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8f795f5f1a0541e28e28c4fc48877888', 'request_id': 'req-e04f3d59-3ea8-4963-aa8d-c50c68766da3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:04:57.059605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e04f3d59-3ea8-4963-aa8d-c50c68766da3 admin admin] The token cache is being invalidate because project 8f795f5f1a0541e28e28c4fc48877888 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:04:57.059892 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e04f3d59-3ea8-4963-aa8d-c50c68766da3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:04:57.060519 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e04f3d59-3ea8-4963-aa8d-c50c68766da3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:04:57.062028 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1168/2348] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:04:56 2022] DELETE /identity/v3/projects/8f795f5f1a0541e28e28c4fc48877888 => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:04:57.067735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6accde21-c7a4-405e-b1bb-511406f923c9 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:57.068192 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6accde21-c7a4-405e-b1bb-511406f923c9 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:57.068307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6accde21-c7a4-405e-b1bb-511406f923c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:57.088870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-6accde21-c7a4-405e-b1bb-511406f923c9 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:04:57.090093 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-6accde21-c7a4-405e-b1bb-511406f923c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:57.147890 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1181/2349] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:04:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:04:57.155610 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f493f921-8ffa-4fab-a885-d32ad7cf2f4f None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:57.155890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f493f921-8ffa-4fab-a885-d32ad7cf2f4f None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:57.156159 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f493f921-8ffa-4fab-a885-d32ad7cf2f4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:57.179174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-f493f921-8ffa-4fab-a885-d32ad7cf2f4f None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:04:57.180380 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f493f921-8ffa-4fab-a885-d32ad7cf2f4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:57.188796 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1169/2350] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:04:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:04:57.193783 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6accde21-c7a4-405e-b1bb-511406f923c9 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:57.204600 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6accde21-c7a4-405e-b1bb-511406f923c9 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:57.209303 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9367c23b-f06d-46a4-93da-93db32002e33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:57.209914 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9367c23b-f06d-46a4-93da-93db32002e33 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:57.210194 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9367c23b-f06d-46a4-93da-93db32002e33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:57.210622 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9367c23b-f06d-46a4-93da-93db32002e33 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:57.211282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9367c23b-f06d-46a4-93da-93db32002e33 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:04:57.211588 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9367c23b-f06d-46a4-93da-93db32002e33 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:57.213357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9367c23b-f06d-46a4-93da-93db32002e33 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:57.225937 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1182/2351] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:04:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:04:57.230229 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f493f921-8ffa-4fab-a885-d32ad7cf2f4f None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:57.240149 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f493f921-8ffa-4fab-a885-d32ad7cf2f4f None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:57.244858 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21174d4-7df6-44b5-b465-87d65a7015f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:57.245488 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21174d4-7df6-44b5-b465-87d65a7015f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:57.245770 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21174d4-7df6-44b5-b465-87d65a7015f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:57.246043 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21174d4-7df6-44b5-b465-87d65a7015f8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:57.246712 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21174d4-7df6-44b5-b465-87d65a7015f8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:57.248591 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21174d4-7df6-44b5-b465-87d65a7015f8 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:57.344743 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1170/2352] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:04:57 2022] POST /identity/v3/projects => generated 387 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:04:57.348726 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9367c23b-f06d-46a4-93da-93db32002e33 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:57.359411 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9367c23b-f06d-46a4-93da-93db32002e33 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:57.417247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a116b72-f694-47fc-afa9-62de29d0dc76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:57.417847 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a116b72-f694-47fc-afa9-62de29d0dc76 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:57.418133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a116b72-f694-47fc-afa9-62de29d0dc76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:57.418439 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a116b72-f694-47fc-afa9-62de29d0dc76 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:57.419075 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a116b72-f694-47fc-afa9-62de29d0dc76 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:57.420990 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a116b72-f694-47fc-afa9-62de29d0dc76 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:57.526123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1183/2353] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:04:57 2022] POST /identity/v3/users => generated 294 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:04:57.530534 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21174d4-7df6-44b5-b465-87d65a7015f8 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:57.542100 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21174d4-7df6-44b5-b465-87d65a7015f8 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:57.546693 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbe0b51-e652-42e2-aba1-2bebfb7ea771 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:57.548008 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbe0b51-e652-42e2-aba1-2bebfb7ea771 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:57.548424 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbe0b51-e652-42e2-aba1-2bebfb7ea771 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:57.548639 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbe0b51-e652-42e2-aba1-2bebfb7ea771 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:57.549283 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fbe0b51-e652-42e2-aba1-2bebfb7ea771 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:57.550890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fbe0b51-e652-42e2-aba1-2bebfb7ea771 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:57.560606 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1171/2354] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:04:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:04:57.564544 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a116b72-f694-47fc-afa9-62de29d0dc76 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:57.574455 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a116b72-f694-47fc-afa9-62de29d0dc76 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:57.579092 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6a6060-46dc-48f6-9164-20f3788a18b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:57.579682 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6a6060-46dc-48f6-9164-20f3788a18b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:57.579966 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6a6060-46dc-48f6-9164-20f3788a18b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:57.580241 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6a6060-46dc-48f6-9164-20f3788a18b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:57.580881 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6a6060-46dc-48f6-9164-20f3788a18b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:57.582496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6a6060-46dc-48f6-9164-20f3788a18b2 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:57.593035 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1184/2355] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:04:57 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:04:57.597736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbe0b51-e652-42e2-aba1-2bebfb7ea771 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:57.607424 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbe0b51-e652-42e2-aba1-2bebfb7ea771 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:57.612661 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96deb87-46b6-48b4-bdc5-b5019119c0f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:57.613295 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96deb87-46b6-48b4-bdc5-b5019119c0f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:57.613573 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96deb87-46b6-48b4-bdc5-b5019119c0f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:57.613841 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96deb87-46b6-48b4-bdc5-b5019119c0f3 admin admin] PATH_INFO: `/v3/projects/66f7fb13cacb4e4399a44dc31c4b40d3/users/4e3bc9e17d9143d998486d31c98bd145/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:57.628618 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96deb87-46b6-48b4-bdc5-b5019119c0f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=66f7fb13cacb4e4399a44dc31c4b40d3, user_id=4e3bc9e17d9143d998486d31c98bd145, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:57.631360 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96deb87-46b6-48b4-bdc5-b5019119c0f3 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:57.714253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1172/2356] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:04:57 2022] PUT /identity/v3/projects/66f7fb13cacb4e4399a44dc31c4b40d3/users/4e3bc9e17d9143d998486d31c98bd145/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:04:57.725684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559179ef-f4d8-4e72-89ad-38b0ff6dd3dc None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:57.726018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559179ef-f4d8-4e72-89ad-38b0ff6dd3dc None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:57.726291 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559179ef-f4d8-4e72-89ad-38b0ff6dd3dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:57.751986 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:04:57.754885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-559179ef-f4d8-4e72-89ad-38b0ff6dd3dc None None] MFA Rules not processed for user `4e3bc9e17d9143d998486d31c98bd145`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:04:57.756091 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-559179ef-f4d8-4e72-89ad-38b0ff6dd3dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:57.819107 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1185/2357] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:04:57 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:04:57.830455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96deb87-46b6-48b4-bdc5-b5019119c0f3 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:04:57.832270 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-b96deb87-46b6-48b4-bdc5-b5019119c0f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:04:57.890158 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96deb87-46b6-48b4-bdc5-b5019119c0f3 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:04:57.893943 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30bbbc3-3c07-4e0d-903b-8c29ebd24df0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:04:57.894573 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30bbbc3-3c07-4e0d-903b-8c29ebd24df0 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:04:57.894853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30bbbc3-3c07-4e0d-903b-8c29ebd24df0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:04:57.895121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30bbbc3-3c07-4e0d-903b-8c29ebd24df0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:04:57.903192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30bbbc3-3c07-4e0d-903b-8c29ebd24df0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:04:57.904600 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30bbbc3-3c07-4e0d-903b-8c29ebd24df0 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:04:57.913254 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1173/2358] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:04:57 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:05:07.817834 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=26 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:05:07.909326 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:DELETE=4,keystone:SELECT=60,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:05:18.854707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559179ef-f4d8-4e72-89ad-38b0ff6dd3dc None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:18.916074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559179ef-f4d8-4e72-89ad-38b0ff6dd3dc None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:18.920812 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9642177c-47a1-4c46-8858-a0ccecfc991c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:18.921427 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9642177c-47a1-4c46-8858-a0ccecfc991c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:18.921707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9642177c-47a1-4c46-8858-a0ccecfc991c admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:18.921978 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9642177c-47a1-4c46-8858-a0ccecfc991c admin admin] PATH_INFO: `/v3/users/4e3bc9e17d9143d998486d31c98bd145` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:18.923115 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9642177c-47a1-4c46-8858-a0ccecfc991c admin admin] RBAC: Authorizing `identity:delete_user(user_id=4e3bc9e17d9143d998486d31c98bd145)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:18.925025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9642177c-47a1-4c46-8858-a0ccecfc991c admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:19.079864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9642177c-47a1-4c46-8858-a0ccecfc991c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4e3bc9e17d9143d998486d31c98bd145'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:05:19.151406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9642177c-47a1-4c46-8858-a0ccecfc991c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4e3bc9e17d9143d998486d31c98bd145'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:05:19.164055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9642177c-47a1-4c46-8858-a0ccecfc991c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4e3bc9e17d9143d998486d31c98bd145'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:05:19.164680 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9642177c-47a1-4c46-8858-a0ccecfc991c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4e3bc9e17d9143d998486d31c98bd145'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:05:19.171669 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-9642177c-47a1-4c46-8858-a0ccecfc991c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4e3bc9e17d9143d998486d31c98bd145'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:05:19.178438 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1186/2359] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:05:18 2022] DELETE /identity/v3/users/4e3bc9e17d9143d998486d31c98bd145 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:05:19.183590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b272bb3d-a365-4301-b35f-e25315daefe5 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:19.184076 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b272bb3d-a365-4301-b35f-e25315daefe5 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:19.184171 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b272bb3d-a365-4301-b35f-e25315daefe5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:19.203027 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-b272bb3d-a365-4301-b35f-e25315daefe5 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:05:19.207022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-b272bb3d-a365-4301-b35f-e25315daefe5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:05:19.268173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1174/2360] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:05:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:05:19.278251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9642177c-47a1-4c46-8858-a0ccecfc991c admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:19.279672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-9642177c-47a1-4c46-8858-a0ccecfc991c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:05:19.337307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9642177c-47a1-4c46-8858-a0ccecfc991c admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:19.340924 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0cba50-eefd-4f87-b899-179c8cdba6fe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:19.341517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0cba50-eefd-4f87-b899-179c8cdba6fe service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:19.341794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0cba50-eefd-4f87-b899-179c8cdba6fe service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:19.342062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0cba50-eefd-4f87-b899-179c8cdba6fe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:19.351878 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0cba50-eefd-4f87-b899-179c8cdba6fe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:19.353294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0cba50-eefd-4f87-b899-179c8cdba6fe service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:19.363000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1187/2361] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:05:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:05:19.796617 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b272bb3d-a365-4301-b35f-e25315daefe5 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:19.798047 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-b272bb3d-a365-4301-b35f-e25315daefe5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:05:19.810532 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b272bb3d-a365-4301-b35f-e25315daefe5 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:19.814801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06686dd7-8f9b-4d04-b1e1-615e253bbba7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:19.815435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06686dd7-8f9b-4d04-b1e1-615e253bbba7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:19.815766 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06686dd7-8f9b-4d04-b1e1-615e253bbba7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:19.816034 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06686dd7-8f9b-4d04-b1e1-615e253bbba7 admin admin] PATH_INFO: `/v3/projects/66f7fb13cacb4e4399a44dc31c4b40d3` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:19.817178 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06686dd7-8f9b-4d04-b1e1-615e253bbba7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=66f7fb13cacb4e4399a44dc31c4b40d3)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:19.819381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06686dd7-8f9b-4d04-b1e1-615e253bbba7 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:19.934738 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-06686dd7-8f9b-4d04-b1e1-615e253bbba7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '66f7fb13cacb4e4399a44dc31c4b40d3', 'request_id': 'req-06686dd7-8f9b-4d04-b1e1-615e253bbba7'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:05:20.049089 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-06686dd7-8f9b-4d04-b1e1-615e253bbba7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '66f7fb13cacb4e4399a44dc31c4b40d3', 'request_id': 'req-06686dd7-8f9b-4d04-b1e1-615e253bbba7'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:05:20.055198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-06686dd7-8f9b-4d04-b1e1-615e253bbba7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '66f7fb13cacb4e4399a44dc31c4b40d3', 'request_id': 'req-06686dd7-8f9b-4d04-b1e1-615e253bbba7'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:05:20.066179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-06686dd7-8f9b-4d04-b1e1-615e253bbba7 admin admin] The token cache is being invalidate because project 66f7fb13cacb4e4399a44dc31c4b40d3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:05:20.066566 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-06686dd7-8f9b-4d04-b1e1-615e253bbba7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:05:20.067284 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-06686dd7-8f9b-4d04-b1e1-615e253bbba7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:05:20.069475 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1175/2362] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:05:19 2022] DELETE /identity/v3/projects/66f7fb13cacb4e4399a44dc31c4b40d3 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:05:20.075694 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1184d1f-6b1e-4db5-82bc-8f0b08c1e02f None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:20.076179 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1184d1f-6b1e-4db5-82bc-8f0b08c1e02f None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:20.076297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1184d1f-6b1e-4db5-82bc-8f0b08c1e02f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:20.097271 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c1184d1f-6b1e-4db5-82bc-8f0b08c1e02f None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:05:20.098624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c1184d1f-6b1e-4db5-82bc-8f0b08c1e02f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:05:20.115783 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:05:20.158669 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1188/2363] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:05:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:05:20.166906 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca311c59-50a5-47f7-a06e-6ef28b1e0d4b None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:20.167358 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca311c59-50a5-47f7-a06e-6ef28b1e0d4b None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:20.167613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca311c59-50a5-47f7-a06e-6ef28b1e0d4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:20.187375 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-ca311c59-50a5-47f7-a06e-6ef28b1e0d4b None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:05:20.188564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ca311c59-50a5-47f7-a06e-6ef28b1e0d4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:05:20.197913 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1176/2364] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:05:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:05:20.202495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1184d1f-6b1e-4db5-82bc-8f0b08c1e02f None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:20.213062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1184d1f-6b1e-4db5-82bc-8f0b08c1e02f None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:20.218369 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712fb0b5-855a-450a-a800-30e8e3841ea5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:20.219009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712fb0b5-855a-450a-a800-30e8e3841ea5 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:20.219283 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712fb0b5-855a-450a-a800-30e8e3841ea5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:20.219551 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712fb0b5-855a-450a-a800-30e8e3841ea5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:20.220273 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712fb0b5-855a-450a-a800-30e8e3841ea5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:05:20.220597 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712fb0b5-855a-450a-a800-30e8e3841ea5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:20.222215 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712fb0b5-855a-450a-a800-30e8e3841ea5 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:20.234319 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1189/2365] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:05:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:05:20.238895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca311c59-50a5-47f7-a06e-6ef28b1e0d4b None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:20.248816 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca311c59-50a5-47f7-a06e-6ef28b1e0d4b None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:20.253444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d996e4f8-bff0-452a-b89e-6449be645bcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:20.254036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d996e4f8-bff0-452a-b89e-6449be645bcd admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:20.254315 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d996e4f8-bff0-452a-b89e-6449be645bcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:20.254626 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d996e4f8-bff0-452a-b89e-6449be645bcd admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:20.255253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d996e4f8-bff0-452a-b89e-6449be645bcd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:20.258024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d996e4f8-bff0-452a-b89e-6449be645bcd admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:20.336124 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1177/2366] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:05:20 2022] POST /identity/v3/projects => generated 373 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:05:20.340497 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712fb0b5-855a-450a-a800-30e8e3841ea5 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:20.350855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712fb0b5-855a-450a-a800-30e8e3841ea5 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:20.406346 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9144763e-6533-4cf9-8c0b-67d4e4d23188 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:20.406949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9144763e-6533-4cf9-8c0b-67d4e4d23188 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:20.407229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9144763e-6533-4cf9-8c0b-67d4e4d23188 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:20.407499 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9144763e-6533-4cf9-8c0b-67d4e4d23188 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:20.408120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9144763e-6533-4cf9-8c0b-67d4e4d23188 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:20.410026 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9144763e-6533-4cf9-8c0b-67d4e4d23188 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:20.531546 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1190/2367] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:05:20 2022] POST /identity/v3/users => generated 287 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:05:20.535739 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d996e4f8-bff0-452a-b89e-6449be645bcd admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:20.546155 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d996e4f8-bff0-452a-b89e-6449be645bcd admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:20.550728 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fccec9-45ca-44d8-8985-21a5bcd85b03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:20.551328 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fccec9-45ca-44d8-8985-21a5bcd85b03 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:20.551613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fccec9-45ca-44d8-8985-21a5bcd85b03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:20.551885 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fccec9-45ca-44d8-8985-21a5bcd85b03 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:20.553369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fccec9-45ca-44d8-8985-21a5bcd85b03 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:20.555025 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fccec9-45ca-44d8-8985-21a5bcd85b03 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:20.565104 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1178/2368] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:05:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:05:20.569658 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9144763e-6533-4cf9-8c0b-67d4e4d23188 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:20.579845 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9144763e-6533-4cf9-8c0b-67d4e4d23188 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:20.584660 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5a967b-daad-4787-b228-9bd8ee046db6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:20.585249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5a967b-daad-4787-b228-9bd8ee046db6 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:20.596935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5a967b-daad-4787-b228-9bd8ee046db6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:20.596935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5a967b-daad-4787-b228-9bd8ee046db6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:20.596935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5a967b-daad-4787-b228-9bd8ee046db6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:20.596935 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5a967b-daad-4787-b228-9bd8ee046db6 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:20.600229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1191/2369] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:05:20 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:05:20.604568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fccec9-45ca-44d8-8985-21a5bcd85b03 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:20.614611 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fccec9-45ca-44d8-8985-21a5bcd85b03 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:20.619912 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128ea88a-3a87-429c-9f8b-ebeb7f99b9b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:20.624161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128ea88a-3a87-429c-9f8b-ebeb7f99b9b5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:20.624161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128ea88a-3a87-429c-9f8b-ebeb7f99b9b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:20.624161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128ea88a-3a87-429c-9f8b-ebeb7f99b9b5 admin admin] PATH_INFO: `/v3/projects/728460a287914347b74cfe886f56ee6c/users/4ad2b6e8ec52485488f27ea84f2c69cc/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:20.636202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128ea88a-3a87-429c-9f8b-ebeb7f99b9b5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=728460a287914347b74cfe886f56ee6c, user_id=4ad2b6e8ec52485488f27ea84f2c69cc, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:20.638869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128ea88a-3a87-429c-9f8b-ebeb7f99b9b5 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:20.701563 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1179/2370] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:05:20 2022] PUT /identity/v3/projects/728460a287914347b74cfe886f56ee6c/users/4ad2b6e8ec52485488f27ea84f2c69cc/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:05:20.706834 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce290c9f-b5b4-4f39-98f2-928d0c021cd5 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:20.707122 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce290c9f-b5b4-4f39-98f2-928d0c021cd5 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:20.707393 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce290c9f-b5b4-4f39-98f2-928d0c021cd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:20.727138 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-ce290c9f-b5b4-4f39-98f2-928d0c021cd5 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:05:20.728342 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-ce290c9f-b5b4-4f39-98f2-928d0c021cd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:05:20.791738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1192/2371] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:05:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:05:20.801574 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128ea88a-3a87-429c-9f8b-ebeb7f99b9b5 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:20.803297 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-128ea88a-3a87-429c-9f8b-ebeb7f99b9b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:05:20.826108 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:05:20.860519 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128ea88a-3a87-429c-9f8b-ebeb7f99b9b5 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:20.864130 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ca4650-ecbc-42c4-8122-f9ee8961309a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:20.864723 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ca4650-ecbc-42c4-8122-f9ee8961309a service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:20.865005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ca4650-ecbc-42c4-8122-f9ee8961309a service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:20.865344 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ca4650-ecbc-42c4-8122-f9ee8961309a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:20.874313 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ca4650-ecbc-42c4-8122-f9ee8961309a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:20.876217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ca4650-ecbc-42c4-8122-f9ee8961309a service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:20.886192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1180/2372] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:05:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:05:26.289029 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d793149e-82cb-4b58-a387-6dd3b8ce18d3 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:26.289029 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d793149e-82cb-4b58-a387-6dd3b8ce18d3 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:26.289780 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d793149e-82cb-4b58-a387-6dd3b8ce18d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:26.321912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-d793149e-82cb-4b58-a387-6dd3b8ce18d3 None None] MFA Rules not processed for user `4ad2b6e8ec52485488f27ea84f2c69cc`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:05:26.323468 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d793149e-82cb-4b58-a387-6dd3b8ce18d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:05:26.343486 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:05:26.394069 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1193/2373] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:05:26 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:05:26.399166 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ca4650-ecbc-42c4-8122-f9ee8961309a service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:26.409230 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ca4650-ecbc-42c4-8122-f9ee8961309a service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:26.415257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7b479e-6766-4438-abc4-aa30e7b3c333 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:26.415995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7b479e-6766-4438-abc4-aa30e7b3c333 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:26.416346 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7b479e-6766-4438-abc4-aa30e7b3c333 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:26.416626 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7b479e-6766-4438-abc4-aa30e7b3c333 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:26.417281 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7b479e-6766-4438-abc4-aa30e7b3c333 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:26.419261 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7b479e-6766-4438-abc4-aa30e7b3c333 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:26.505336 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1181/2374] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:05:26 2022] POST /identity/v3/projects => generated 371 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:05:26.509803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d793149e-82cb-4b58-a387-6dd3b8ce18d3 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:26.520987 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d793149e-82cb-4b58-a387-6dd3b8ce18d3 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:26.579110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da1155a-c896-4a80-b6e4-69d2b82ce0ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:26.579717 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da1155a-c896-4a80-b6e4-69d2b82ce0ef admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:26.579994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da1155a-c896-4a80-b6e4-69d2b82ce0ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:26.580263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da1155a-c896-4a80-b6e4-69d2b82ce0ef admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:26.580899 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da1155a-c896-4a80-b6e4-69d2b82ce0ef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:26.582849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da1155a-c896-4a80-b6e4-69d2b82ce0ef admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:26.680962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1194/2375] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:05:26 2022] POST /identity/v3/users => generated 292 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:05:26.684821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7b479e-6766-4438-abc4-aa30e7b3c333 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:26.695678 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7b479e-6766-4438-abc4-aa30e7b3c333 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:26.700241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a8ecc9-2082-47f0-be48-17eac5495bc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:26.700842 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a8ecc9-2082-47f0-be48-17eac5495bc8 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:26.701118 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a8ecc9-2082-47f0-be48-17eac5495bc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:26.701388 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a8ecc9-2082-47f0-be48-17eac5495bc8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:26.702062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a8ecc9-2082-47f0-be48-17eac5495bc8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:26.703674 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18a8ecc9-2082-47f0-be48-17eac5495bc8 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:26.719274 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1182/2376] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:05:26 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:05:26.720673 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da1155a-c896-4a80-b6e4-69d2b82ce0ef admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:26.731037 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da1155a-c896-4a80-b6e4-69d2b82ce0ef admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:26.735691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65aba128-5bf3-4e20-8418-835518c681b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:26.736342 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65aba128-5bf3-4e20-8418-835518c681b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:26.736621 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65aba128-5bf3-4e20-8418-835518c681b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:26.736892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65aba128-5bf3-4e20-8418-835518c681b0 admin admin] PATH_INFO: `/v3/projects/09f3cfa0c4164cb0b07f49311387aa98/users/4fa66c39478e46ef9c69e3c1ad446252/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:26.752465 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65aba128-5bf3-4e20-8418-835518c681b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=09f3cfa0c4164cb0b07f49311387aa98, user_id=4fa66c39478e46ef9c69e3c1ad446252, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:26.755174 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65aba128-5bf3-4e20-8418-835518c681b0 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:26.869709 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1195/2377] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:05:26 2022] PUT /identity/v3/projects/09f3cfa0c4164cb0b07f49311387aa98/users/4fa66c39478e46ef9c69e3c1ad446252/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:05:26.873719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a8ecc9-2082-47f0-be48-17eac5495bc8 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:26.884246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a8ecc9-2082-47f0-be48-17eac5495bc8 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:26.939745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5372d4-e6c4-47f7-9d84-56ac00056e23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:26.940268 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5372d4-e6c4-47f7-9d84-56ac00056e23 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:26.940549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5372d4-e6c4-47f7-9d84-56ac00056e23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:26.940820 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5372d4-e6c4-47f7-9d84-56ac00056e23 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:26.941453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5372d4-e6c4-47f7-9d84-56ac00056e23 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:26.943072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5372d4-e6c4-47f7-9d84-56ac00056e23 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:26.953318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1183/2378] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:05:26 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:05:26.957616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65aba128-5bf3-4e20-8418-835518c681b0 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:26.968884 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65aba128-5bf3-4e20-8418-835518c681b0 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:26.973767 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c7220b-4eb0-4f93-8456-11f4e300f9d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:26.974458 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c7220b-4eb0-4f93-8456-11f4e300f9d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:26.974744 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c7220b-4eb0-4f93-8456-11f4e300f9d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:26.975017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c7220b-4eb0-4f93-8456-11f4e300f9d3 admin admin] PATH_INFO: `/v3/projects/09f3cfa0c4164cb0b07f49311387aa98/users/4fa66c39478e46ef9c69e3c1ad446252/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:26.977130 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c7220b-4eb0-4f93-8456-11f4e300f9d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=09f3cfa0c4164cb0b07f49311387aa98, user_id=4fa66c39478e46ef9c69e3c1ad446252, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:26.979844 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c7220b-4eb0-4f93-8456-11f4e300f9d3 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:26.990041 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1196/2379] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:05:26 2022] PUT /identity/v3/projects/09f3cfa0c4164cb0b07f49311387aa98/users/4fa66c39478e46ef9c69e3c1ad446252/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:05:31.455783 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5372d4-e6c4-47f7-9d84-56ac00056e23 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:31.459775 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-fa5372d4-e6c4-47f7-9d84-56ac00056e23 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:05:31.613854 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5372d4-e6c4-47f7-9d84-56ac00056e23 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:31.681854 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:05:31.778604 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd55f033-c98e-488a-966d-bdba238472f7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce249d4a99c34891b6ba941f57a14dd4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:31.779257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd55f033-c98e-488a-966d-bdba238472f7 service placement] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:31.779538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd55f033-c98e-488a-966d-bdba238472f7 service placement] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:31.779810 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd55f033-c98e-488a-966d-bdba238472f7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:31.785472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-cd55f033-c98e-488a-966d-bdba238472f7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:05:31.816847 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd55f033-c98e-488a-966d-bdba238472f7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:31.818837 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd55f033-c98e-488a-966d-bdba238472f7 service placement] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:31.842932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1184/2380] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:05:31 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:05:33.862565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c14de1c-3794-4040-ac45-a3973109da9e None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:33.863073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c14de1c-3794-4040-ac45-a3973109da9e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:33.877396 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c14de1c-3794-4040-ac45-a3973109da9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:33.923770 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-5c14de1c-3794-4040-ac45-a3973109da9e None None] MFA Rules not processed for user `4fa66c39478e46ef9c69e3c1ad446252`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:05:33.930664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-5c14de1c-3794-4040-ac45-a3973109da9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:05:34.023200 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddcfcd4-9ea4-4c30-b792-e3f5bff777a1 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:34.025541 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddcfcd4-9ea4-4c30-b792-e3f5bff777a1 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:34.025932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddcfcd4-9ea4-4c30-b792-e3f5bff777a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:34.051332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:05:34.068218 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1197/2381] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:05:33 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:05:34.075490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-5ddcfcd4-9ea4-4c30-b792-e3f5bff777a1 None None] MFA Rules not processed for user `b471d768cd814ff8b4f6bd35d21e24a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:05:34.076965 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-5ddcfcd4-9ea4-4c30-b792-e3f5bff777a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:05:34.082035 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c14de1c-3794-4040-ac45-a3973109da9e None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:34.083707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-5c14de1c-3794-4040-ac45-a3973109da9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:05:34.173879 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1185/2382] 10.222.0.62 () {58 vars in 1009 bytes} [Fri Sep 23 14:05:34 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:05:34.187953 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddcfcd4-9ea4-4c30-b792-e3f5bff777a1 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:34.190016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-5ddcfcd4-9ea4-4c30-b792-e3f5bff777a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:05:34.208490 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c14de1c-3794-4040-ac45-a3973109da9e None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:34.222161 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d6e18e-ade3-4cc7-b068-c7fc378d4f7b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:34.222921 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d6e18e-ade3-4cc7-b068-c7fc378d4f7b service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:34.223166 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d6e18e-ade3-4cc7-b068-c7fc378d4f7b service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:34.223439 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d6e18e-ade3-4cc7-b068-c7fc378d4f7b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:34.281488 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddcfcd4-9ea4-4c30-b792-e3f5bff777a1 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:34.287632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d5c7dd-41b1-448d-8641-a7d6ec257c47 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b9269509f1d454eab7a3008c2ba0039', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:34.288932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d5c7dd-41b1-448d-8641-a7d6ec257c47 service swift] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:34.289220 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d5c7dd-41b1-448d-8641-a7d6ec257c47 service swift] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:34.289492 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d5c7dd-41b1-448d-8641-a7d6ec257c47 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:34.299277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d5c7dd-41b1-448d-8641-a7d6ec257c47 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:34.300937 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d6e18e-ade3-4cc7-b068-c7fc378d4f7b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:34.302437 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d5c7dd-41b1-448d-8641-a7d6ec257c47 service swift] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:34.302880 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d6e18e-ade3-4cc7-b068-c7fc378d4f7b service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:34.315987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1186/2383] 10.222.0.62 () {60 vars in 1384 bytes} [Fri Sep 23 14:05:34 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 23 14:05:34.318644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1198/2384] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:05:34 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 237 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:05:38.138900 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1f8f63-cd4a-45cc-b308-f102d3bba365 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:38.138900 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1f8f63-cd4a-45cc-b308-f102d3bba365 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:38.138900 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1f8f63-cd4a-45cc-b308-f102d3bba365 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:38.175715 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-ac1f8f63-cd4a-45cc-b308-f102d3bba365 None None] MFA Rules not processed for user `4ad2b6e8ec52485488f27ea84f2c69cc`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:05:38.177572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ac1f8f63-cd4a-45cc-b308-f102d3bba365 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:05:38.190468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1187/2385] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:05:38 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:05:45.108330 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=58 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:05:48.167107 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=79 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:05:54.208511 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfd3ba5-30d2-4158-9da8-a8f02e63c8c2 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:54.209092 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfd3ba5-30d2-4158-9da8-a8f02e63c8c2 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:54.209092 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adfd3ba5-30d2-4158-9da8-a8f02e63c8c2 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:54.214314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1199/2386] 10.222.0.62 () {54 vars in 921 bytes} [Fri Sep 23 14:05:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:05:54.217056 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cef4bbd-fd16-4794-a781-e787bd50bf4f None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:54.217484 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cef4bbd-fd16-4794-a781-e787bd50bf4f None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:54.217578 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cef4bbd-fd16-4794-a781-e787bd50bf4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:54.237419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-7cef4bbd-fd16-4794-a781-e787bd50bf4f None None] MFA Rules not processed for user `f01ed94dbb384a178ad0a45a6f3d8c17`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:05:54.238671 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-7cef4bbd-fd16-4794-a781-e787bd50bf4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:05:54.249185 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1188/2387] 10.222.0.62 () {58 vars in 1018 bytes} [Fri Sep 23 14:05:54 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:05:54.274904 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adfd3ba5-30d2-4158-9da8-a8f02e63c8c2 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:54.277360 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-adfd3ba5-30d2-4158-9da8-a8f02e63c8c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:05:54.337962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adfd3ba5-30d2-4158-9da8-a8f02e63c8c2 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:54.341633 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467def96-e956-4d77-8a3d-bfcda1264f93 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'acdf4a7bd608492390176dcc42ca1267', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:54.342230 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467def96-e956-4d77-8a3d-bfcda1264f93 service glance] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:54.342566 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467def96-e956-4d77-8a3d-bfcda1264f93 service glance] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:54.342835 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467def96-e956-4d77-8a3d-bfcda1264f93 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:54.352701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467def96-e956-4d77-8a3d-bfcda1264f93 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:54.354966 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467def96-e956-4d77-8a3d-bfcda1264f93 service glance] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:54.365692 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1200/2388] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:05:54 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:05:54.568242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398d6cef-50b7-4f5b-926f-2e9997217a0d None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:54.569082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398d6cef-50b7-4f5b-926f-2e9997217a0d None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:54.569082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398d6cef-50b7-4f5b-926f-2e9997217a0d None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:54.570166 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1189/2389] 10.222.0.62 () {54 vars in 921 bytes} [Fri Sep 23 14:05:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:05:54.575140 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a92f92-317f-4a40-ba9a-382ba85360b2 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:54.575425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a92f92-317f-4a40-ba9a-382ba85360b2 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:54.575694 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a92f92-317f-4a40-ba9a-382ba85360b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:54.596516 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-10a92f92-317f-4a40-ba9a-382ba85360b2 None None] MFA Rules not processed for user `580a35627c234a25aa04f5d376177ca5`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:05:54.597643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-10a92f92-317f-4a40-ba9a-382ba85360b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:05:54.650694 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1201/2390] 10.222.0.62 () {58 vars in 1018 bytes} [Fri Sep 23 14:05:54 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:05:54.662961 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398d6cef-50b7-4f5b-926f-2e9997217a0d None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:54.674952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398d6cef-50b7-4f5b-926f-2e9997217a0d None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:54.678599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7536cf39-ed2a-4ab5-952f-a12be407ca46 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:54.679215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7536cf39-ed2a-4ab5-952f-a12be407ca46 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:54.679493 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7536cf39-ed2a-4ab5-952f-a12be407ca46 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:54.679769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7536cf39-ed2a-4ab5-952f-a12be407ca46 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:54.688218 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7536cf39-ed2a-4ab5-952f-a12be407ca46 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:54.689655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7536cf39-ed2a-4ab5-952f-a12be407ca46 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:54.698708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1190/2391] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:05:54 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:05:54.789959 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf04c64-62e3-4ef1-9410-d424cc8739b3 None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:54.790380 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf04c64-62e3-4ef1-9410-d424cc8739b3 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:54.790835 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf04c64-62e3-4ef1-9410-d424cc8739b3 None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:54.793027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1202/2392] 10.222.0.62 () {54 vars in 921 bytes} [Fri Sep 23 14:05:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:05:54.799642 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52f6954-3bd2-42bd-ad71-1c8528fa81d5 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:54.799974 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52f6954-3bd2-42bd-ad71-1c8528fa81d5 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:54.800340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52f6954-3bd2-42bd-ad71-1c8528fa81d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:54.820865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-e52f6954-3bd2-42bd-ad71-1c8528fa81d5 None None] MFA Rules not processed for user `f01ed94dbb384a178ad0a45a6f3d8c17`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:05:54.822100 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e52f6954-3bd2-42bd-ad71-1c8528fa81d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:05:54.831720 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1191/2393] 10.222.0.62 () {58 vars in 1018 bytes} [Fri Sep 23 14:05:54 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:05:54.843855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf04c64-62e3-4ef1-9410-d424cc8739b3 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:05:54.856455 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf04c64-62e3-4ef1-9410-d424cc8739b3 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:05:54.860538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd9ae08-4a07-4409-8280-07e3b990df06 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:05:54.861238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd9ae08-4a07-4409-8280-07e3b990df06 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:54.861520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd9ae08-4a07-4409-8280-07e3b990df06 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:54.861789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd9ae08-4a07-4409-8280-07e3b990df06 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:54.872835 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd9ae08-4a07-4409-8280-07e3b990df06 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:05:54.874301 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd9ae08-4a07-4409-8280-07e3b990df06 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:05:54.885517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1203/2394] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:05:54 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:05:56.288381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf558249-f646-4a65-ac75-87f829f24c7c None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:05:56.288807 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf558249-f646-4a65-ac75-87f829f24c7c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:05:56.288945 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf558249-f646-4a65-ac75-87f829f24c7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:05:56.309922 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-bf558249-f646-4a65-ac75-87f829f24c7c None None] MFA Rules not processed for user `0db30f990aae49fabf9b8b19cd019de9`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:05:56.311306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-bf558249-f646-4a65-ac75-87f829f24c7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:05:56.369138 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1192/2395] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:05:56 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:06:01.366957 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa049f35-d5f7-4071-be1e-9c5b581a2754 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:06:01.367563 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa049f35-d5f7-4071-be1e-9c5b581a2754 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:06:01.367563 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa049f35-d5f7-4071-be1e-9c5b581a2754 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:06:01.387101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-aa049f35-d5f7-4071-be1e-9c5b581a2754 None None] MFA Rules not processed for user `28b64a35d7b44519a1d93919f2e23083`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:06:01.388458 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-aa049f35-d5f7-4071-be1e-9c5b581a2754 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:06:01.455458 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1204/2396] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:06:01 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:06:01.757880 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf07153-33f4-4906-8e93-25f893e869d4 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:06:01.758558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf07153-33f4-4906-8e93-25f893e869d4 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:06:01.758899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf07153-33f4-4906-8e93-25f893e869d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:06:01.785724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-3cf07153-33f4-4906-8e93-25f893e869d4 None None] MFA Rules not processed for user `b471d768cd814ff8b4f6bd35d21e24a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:06:01.787012 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3cf07153-33f4-4906-8e93-25f893e869d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:06:01.794985 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1193/2397] 10.222.0.62 () {58 vars in 1009 bytes} [Fri Sep 23 14:06:01 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:06:01.819645 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa049f35-d5f7-4071-be1e-9c5b581a2754 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:06:01.839791 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa049f35-d5f7-4071-be1e-9c5b581a2754 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:06:01.848360 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e809801-f3a4-4dd8-b2a6-b9cbfeee6d98 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b9269509f1d454eab7a3008c2ba0039', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:06:01.849115 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e809801-f3a4-4dd8-b2a6-b9cbfeee6d98 service swift] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:06:01.849511 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e809801-f3a4-4dd8-b2a6-b9cbfeee6d98 service swift] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:06:01.849767 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e809801-f3a4-4dd8-b2a6-b9cbfeee6d98 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:06:01.861315 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e809801-f3a4-4dd8-b2a6-b9cbfeee6d98 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:06:01.864199 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e809801-f3a4-4dd8-b2a6-b9cbfeee6d98 service swift] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:06:01.883115 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1205/2398] 10.222.0.62 () {60 vars in 1384 bytes} [Fri Sep 23 14:06:01 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 23 14:06:11.779128 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=40 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:06:11.878140 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=86 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:06:25.755012 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0f1596-ad16-4e74-9aff-29d145ee98aa None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:06:25.755463 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0f1596-ad16-4e74-9aff-29d145ee98aa None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:06:25.755575 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0f1596-ad16-4e74-9aff-29d145ee98aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:06:25.775643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-5f0f1596-ad16-4e74-9aff-29d145ee98aa None None] MFA Rules not processed for user `b471d768cd814ff8b4f6bd35d21e24a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:06:25.776884 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-5f0f1596-ad16-4e74-9aff-29d145ee98aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:06:25.785981 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1194/2399] 10.222.0.62 () {58 vars in 967 bytes} [Fri Sep 23 14:06:25 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:06:25.799359 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e809801-f3a4-4dd8-b2a6-b9cbfeee6d98 service swift] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:06:25.810245 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e809801-f3a4-4dd8-b2a6-b9cbfeee6d98 service swift] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:06:25.815380 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916b0a22-d201-4634-81c9-ac5551166bfb service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b9269509f1d454eab7a3008c2ba0039', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:06:25.815969 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916b0a22-d201-4634-81c9-ac5551166bfb service swift] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:06:25.816249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916b0a22-d201-4634-81c9-ac5551166bfb service swift] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:06:25.816543 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916b0a22-d201-4634-81c9-ac5551166bfb service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:06:25.824914 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916b0a22-d201-4634-81c9-ac5551166bfb service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:06:25.826538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916b0a22-d201-4634-81c9-ac5551166bfb service swift] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:06:25.835299 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1206/2400] 10.222.0.62 () {60 vars in 1384 bytes} [Fri Sep 23 14:06:25 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 23 14:06:33.103897 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0f1596-ad16-4e74-9aff-29d145ee98aa None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:06:33.114163 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0f1596-ad16-4e74-9aff-29d145ee98aa None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:06:33.117825 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b339d1fd-ee1e-4b20-9f4a-ac9a526b607d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:06:33.121853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b339d1fd-ee1e-4b20-9f4a-ac9a526b607d service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:06:33.121853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b339d1fd-ee1e-4b20-9f4a-ac9a526b607d service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:06:33.121853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b339d1fd-ee1e-4b20-9f4a-ac9a526b607d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:06:33.128054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b339d1fd-ee1e-4b20-9f4a-ac9a526b607d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:06:33.129529 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b339d1fd-ee1e-4b20-9f4a-ac9a526b607d service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:06:33.138799 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1195/2401] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:06:33 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:06:35.832204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=6 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:06:43.134713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=9 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:07:02.362475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107a0f67-1d35-44a2-a64c-91a8fa15d28b None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:02.362987 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107a0f67-1d35-44a2-a64c-91a8fa15d28b None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:02.363151 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107a0f67-1d35-44a2-a64c-91a8fa15d28b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:02.384460 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-107a0f67-1d35-44a2-a64c-91a8fa15d28b None None] MFA Rules not processed for user `9359e90c4861470d9cc060abb9c3b203`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:07:02.385760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-107a0f67-1d35-44a2-a64c-91a8fa15d28b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:02.441232 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1207/2402] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:07:02 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:07:08.559151 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b339d1fd-ee1e-4b20-9f4a-ac9a526b607d service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:08.560602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-b339d1fd-ee1e-4b20-9f4a-ac9a526b607d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:08.572760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b339d1fd-ee1e-4b20-9f4a-ac9a526b607d service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:08.577419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcdd686-0897-4b77-9f3f-aa14c0cac65d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:08.578201 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcdd686-0897-4b77-9f3f-aa14c0cac65d service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:08.578576 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcdd686-0897-4b77-9f3f-aa14c0cac65d service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:08.578921 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afcdd686-0897-4b77-9f3f-aa14c0cac65d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:08.580947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-afcdd686-0897-4b77-9f3f-aa14c0cac65d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:08.596533 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afcdd686-0897-4b77-9f3f-aa14c0cac65d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:08.597959 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afcdd686-0897-4b77-9f3f-aa14c0cac65d service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:08.609984 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1196/2403] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:07:08 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:07:09.915417 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107a0f67-1d35-44a2-a64c-91a8fa15d28b None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:09.924974 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107a0f67-1d35-44a2-a64c-91a8fa15d28b None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:09.928795 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e4acb4-4937-44f3-a289-5619cd6e11e3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:09.929395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e4acb4-4937-44f3-a289-5619cd6e11e3 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:09.929673 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e4acb4-4937-44f3-a289-5619cd6e11e3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:09.929941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e4acb4-4937-44f3-a289-5619cd6e11e3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:09.932950 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-06e4acb4-4937-44f3-a289-5619cd6e11e3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:09.947657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e4acb4-4937-44f3-a289-5619cd6e11e3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:09.949823 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e4acb4-4937-44f3-a289-5619cd6e11e3 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:09.961019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1208/2404] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:07:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:07:15.251911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcdd686-0897-4b77-9f3f-aa14c0cac65d service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:15.253496 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-afcdd686-0897-4b77-9f3f-aa14c0cac65d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:15.268643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afcdd686-0897-4b77-9f3f-aa14c0cac65d service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:15.274560 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d204623d-c89c-46c1-8104-4fe6f668baae service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:15.275189 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204623d-c89c-46c1-8104-4fe6f668baae service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:15.275457 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204623d-c89c-46c1-8104-4fe6f668baae service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:15.275731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d204623d-c89c-46c1-8104-4fe6f668baae service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:15.278345 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d204623d-c89c-46c1-8104-4fe6f668baae service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:15.289683 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d204623d-c89c-46c1-8104-4fe6f668baae service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:15.291308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d204623d-c89c-46c1-8104-4fe6f668baae service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:15.301796 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1197/2405] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:07:15 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:07:19.956075 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=31 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:07:20.122155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e4acb4-4937-44f3-a289-5619cd6e11e3 service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:20.123800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-06e4acb4-4937-44f3-a289-5619cd6e11e3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:20.272799 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e4acb4-4937-44f3-a289-5619cd6e11e3 service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:20.276807 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82214bc4-1154-4ca8-b4bc-ec34d91c08b6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:20.278260 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82214bc4-1154-4ca8-b4bc-ec34d91c08b6 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:20.278717 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82214bc4-1154-4ca8-b4bc-ec34d91c08b6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:20.278998 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82214bc4-1154-4ca8-b4bc-ec34d91c08b6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:20.287978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d204623d-c89c-46c1-8104-4fe6f668baae service nova] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:20.289679 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d204623d-c89c-46c1-8104-4fe6f668baae service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:20.290951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-82214bc4-1154-4ca8-b4bc-ec34d91c08b6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:20.306197 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d204623d-c89c-46c1-8104-4fe6f668baae service nova] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:20.307586 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82214bc4-1154-4ca8-b4bc-ec34d91c08b6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:20.310748 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e1d7ca-75fa-45d1-881d-bc515a824bd6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:20.311487 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e1d7ca-75fa-45d1-881d-bc515a824bd6 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:20.312053 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e1d7ca-75fa-45d1-881d-bc515a824bd6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:20.312521 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e1d7ca-75fa-45d1-881d-bc515a824bd6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:20.321681 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82214bc4-1154-4ca8-b4bc-ec34d91c08b6 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:20.327084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e1d7ca-75fa-45d1-881d-bc515a824bd6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:20.328606 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e1d7ca-75fa-45d1-881d-bc515a824bd6 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:20.336420 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1209/2406] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:07:20 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:07:20.347839 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1198/2407] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:07:20 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:07:22.371266 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82214bc4-1154-4ca8-b4bc-ec34d91c08b6 service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:22.381282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82214bc4-1154-4ca8-b4bc-ec34d91c08b6 service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:22.384908 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc5f01b-693b-48b5-b4b4-c77b8ea2a4d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:22.385495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc5f01b-693b-48b5-b4b4-c77b8ea2a4d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:22.385774 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc5f01b-693b-48b5-b4b4-c77b8ea2a4d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:22.386044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc5f01b-693b-48b5-b4b4-c77b8ea2a4d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:22.387941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-bbc5f01b-693b-48b5-b4b4-c77b8ea2a4d4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:22.451140 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc5f01b-693b-48b5-b4b4-c77b8ea2a4d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:22.452488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc5f01b-693b-48b5-b4b4-c77b8ea2a4d4 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:22.462675 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1210/2408] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:07:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:07:25.244395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4bd086-58dd-43e6-aa82-d269c7db846c None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:25.244913 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4bd086-58dd-43e6-aa82-d269c7db846c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:25.245006 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4bd086-58dd-43e6-aa82-d269c7db846c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:25.266854 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-4c4bd086-58dd-43e6-aa82-d269c7db846c None None] MFA Rules not processed for user `d777c43fe5c142b3b5ea5e2485f4e49f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:07:25.268155 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-4c4bd086-58dd-43e6-aa82-d269c7db846c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:25.323340 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1199/2409] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:07:25 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:07:25.672342 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc5f01b-693b-48b5-b4b4-c77b8ea2a4d4 service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:25.683675 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc5f01b-693b-48b5-b4b4-c77b8ea2a4d4 service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:25.689183 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facd220c-1b9f-4b5b-ac3f-e342ea6794ee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:25.689802 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facd220c-1b9f-4b5b-ac3f-e342ea6794ee service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:25.690077 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facd220c-1b9f-4b5b-ac3f-e342ea6794ee service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:25.690520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facd220c-1b9f-4b5b-ac3f-e342ea6794ee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:25.693665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-facd220c-1b9f-4b5b-ac3f-e342ea6794ee service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:25.704456 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facd220c-1b9f-4b5b-ac3f-e342ea6794ee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:25.706293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facd220c-1b9f-4b5b-ac3f-e342ea6794ee service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:25.717816 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1211/2410] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:07:25 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:07:26.265761 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04772fa1-a688-46b1-99ff-61d27d687d2d None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:26.266272 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04772fa1-a688-46b1-99ff-61d27d687d2d None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:26.266401 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04772fa1-a688-46b1-99ff-61d27d687d2d None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:26.267692 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1200/2411] 10.222.0.62 () {54 vars in 961 bytes} [Fri Sep 23 14:07:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:07:26.273239 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be30230a-2c2a-4c82-867b-379b2aba07d8 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:26.274237 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be30230a-2c2a-4c82-867b-379b2aba07d8 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:26.274237 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be30230a-2c2a-4c82-867b-379b2aba07d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:26.306279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-be30230a-2c2a-4c82-867b-379b2aba07d8 None None] MFA Rules not processed for user `ce249d4a99c34891b6ba941f57a14dd4`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:07:26.307527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-be30230a-2c2a-4c82-867b-379b2aba07d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:26.318083 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1212/2412] 10.222.0.62 () {58 vars in 1058 bytes} [Fri Sep 23 14:07:26 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:07:26.326371 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04772fa1-a688-46b1-99ff-61d27d687d2d None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:26.352525 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04772fa1-a688-46b1-99ff-61d27d687d2d None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:26.358290 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d9c8d6-1447-41a1-a8e1-e365e988f042 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce249d4a99c34891b6ba941f57a14dd4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:26.359034 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d9c8d6-1447-41a1-a8e1-e365e988f042 service placement] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:26.359314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d9c8d6-1447-41a1-a8e1-e365e988f042 service placement] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:26.360335 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d9c8d6-1447-41a1-a8e1-e365e988f042 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:26.362119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-55d9c8d6-1447-41a1-a8e1-e365e988f042 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:26.375399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d9c8d6-1447-41a1-a8e1-e365e988f042 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:26.376980 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d9c8d6-1447-41a1-a8e1-e365e988f042 service placement] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:26.388763 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1201/2413] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:07:26 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:07:27.225442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be30230a-2c2a-4c82-867b-379b2aba07d8 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:27.226920 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-be30230a-2c2a-4c82-867b-379b2aba07d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:27.241719 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be30230a-2c2a-4c82-867b-379b2aba07d8 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:27.246174 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ddadbe0-6608-4493-83df-00a68e8c4d07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:27.246978 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddadbe0-6608-4493-83df-00a68e8c4d07 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:27.247253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddadbe0-6608-4493-83df-00a68e8c4d07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:27.247627 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ddadbe0-6608-4493-83df-00a68e8c4d07 admin admin] PATH_INFO: `/v3/users/28b64a35d7b44519a1d93919f2e23083` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:27.248846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ddadbe0-6608-4493-83df-00a68e8c4d07 admin admin] RBAC: Authorizing `identity:delete_user(user_id=28b64a35d7b44519a1d93919f2e23083)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:27.250822 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ddadbe0-6608-4493-83df-00a68e8c4d07 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:27.467312 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0ddadbe0-6608-4493-83df-00a68e8c4d07 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '28b64a35d7b44519a1d93919f2e23083'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:27.520920 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0ddadbe0-6608-4493-83df-00a68e8c4d07 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '28b64a35d7b44519a1d93919f2e23083'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:27.533894 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0ddadbe0-6608-4493-83df-00a68e8c4d07 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '28b64a35d7b44519a1d93919f2e23083'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:27.534707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0ddadbe0-6608-4493-83df-00a68e8c4d07 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '28b64a35d7b44519a1d93919f2e23083'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:27.535875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:07:27.541925 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0ddadbe0-6608-4493-83df-00a68e8c4d07 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '28b64a35d7b44519a1d93919f2e23083'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:27.548497 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1213/2414] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:07:27 2022] DELETE /identity/v3/users/28b64a35d7b44519a1d93919f2e23083 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:07:27.815146 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d9c8d6-1447-41a1-a8e1-e365e988f042 service placement] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:27.831317 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d9c8d6-1447-41a1-a8e1-e365e988f042 service placement] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:27.893124 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223d80a3-c069-46b6-8d3e-dae30b2a0535 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:27.893752 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223d80a3-c069-46b6-8d3e-dae30b2a0535 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:27.894446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223d80a3-c069-46b6-8d3e-dae30b2a0535 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:27.894446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223d80a3-c069-46b6-8d3e-dae30b2a0535 admin admin] PATH_INFO: `/v3/projects/d08a219027684089a91c354749c67d5d` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:27.895437 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223d80a3-c069-46b6-8d3e-dae30b2a0535 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d08a219027684089a91c354749c67d5d)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:27.897387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223d80a3-c069-46b6-8d3e-dae30b2a0535 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:28.008319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-223d80a3-c069-46b6-8d3e-dae30b2a0535 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd08a219027684089a91c354749c67d5d', 'request_id': 'req-223d80a3-c069-46b6-8d3e-dae30b2a0535'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:28.012947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:07:28.122307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-223d80a3-c069-46b6-8d3e-dae30b2a0535 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd08a219027684089a91c354749c67d5d', 'request_id': 'req-223d80a3-c069-46b6-8d3e-dae30b2a0535'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:28.128321 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-223d80a3-c069-46b6-8d3e-dae30b2a0535 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd08a219027684089a91c354749c67d5d', 'request_id': 'req-223d80a3-c069-46b6-8d3e-dae30b2a0535'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:28.140108 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-223d80a3-c069-46b6-8d3e-dae30b2a0535 admin admin] The token cache is being invalidate because project d08a219027684089a91c354749c67d5d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:07:28.140397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-223d80a3-c069-46b6-8d3e-dae30b2a0535 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:28.141024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-223d80a3-c069-46b6-8d3e-dae30b2a0535 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:28.142559 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1202/2415] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:07:27 2022] DELETE /identity/v3/projects/d08a219027684089a91c354749c67d5d => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:07:28.147811 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec184b3-6df7-4001-8c93-c018da68ed68 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:28.148022 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec184b3-6df7-4001-8c93-c018da68ed68 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:28.148297 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec184b3-6df7-4001-8c93-c018da68ed68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:28.168104 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-0ec184b3-6df7-4001-8c93-c018da68ed68 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:07:28.170225 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-0ec184b3-6df7-4001-8c93-c018da68ed68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:28.254878 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1214/2416] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:07:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:07:28.262513 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbf6780-d3b5-4f11-b225-3854d9a0a2aa None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:28.263011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbf6780-d3b5-4f11-b225-3854d9a0a2aa None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:28.263291 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbf6780-d3b5-4f11-b225-3854d9a0a2aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:28.289866 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-2fbf6780-d3b5-4f11-b225-3854d9a0a2aa None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:07:28.291126 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2fbf6780-d3b5-4f11-b225-3854d9a0a2aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:28.300030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1203/2417] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:07:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:07:28.304702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec184b3-6df7-4001-8c93-c018da68ed68 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:28.315418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec184b3-6df7-4001-8c93-c018da68ed68 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:28.320503 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89910a20-fd3a-414a-9774-d6c8edfbf977 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:28.321114 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89910a20-fd3a-414a-9774-d6c8edfbf977 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:28.322159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89910a20-fd3a-414a-9774-d6c8edfbf977 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:28.322159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89910a20-fd3a-414a-9774-d6c8edfbf977 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:28.322794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89910a20-fd3a-414a-9774-d6c8edfbf977 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:07:28.323097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89910a20-fd3a-414a-9774-d6c8edfbf977 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:28.325104 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89910a20-fd3a-414a-9774-d6c8edfbf977 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:28.338438 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1215/2418] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:07:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:07:28.343320 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbf6780-d3b5-4f11-b225-3854d9a0a2aa None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:28.352765 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fbf6780-d3b5-4f11-b225-3854d9a0a2aa None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:28.364151 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9d50e1-a7f8-4088-888e-8a8d4d589cb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:28.364780 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9d50e1-a7f8-4088-888e-8a8d4d589cb9 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:28.365060 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9d50e1-a7f8-4088-888e-8a8d4d589cb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:28.365333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9d50e1-a7f8-4088-888e-8a8d4d589cb9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:28.365973 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9d50e1-a7f8-4088-888e-8a8d4d589cb9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:28.367920 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9d50e1-a7f8-4088-888e-8a8d4d589cb9 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:28.489850 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89910a20-fd3a-414a-9774-d6c8edfbf977 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:28.505183 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89910a20-fd3a-414a-9774-d6c8edfbf977 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:28.505421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1204/2419] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:07:28 2022] POST /identity/v3/projects => generated 385 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:07:28.575978 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416443df-8d91-4164-8482-64c7fc62874d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:28.576851 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416443df-8d91-4164-8482-64c7fc62874d admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:28.577192 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416443df-8d91-4164-8482-64c7fc62874d admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:28.577523 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416443df-8d91-4164-8482-64c7fc62874d admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:28.578191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416443df-8d91-4164-8482-64c7fc62874d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:28.580197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416443df-8d91-4164-8482-64c7fc62874d admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:28.683416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1216/2420] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:07:28 2022] POST /identity/v3/users => generated 293 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:07:28.687736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9d50e1-a7f8-4088-888e-8a8d4d589cb9 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:28.698024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9d50e1-a7f8-4088-888e-8a8d4d589cb9 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:28.702781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedf77e2-ff26-49ab-80b4-0a819bb3ce49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:28.703420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedf77e2-ff26-49ab-80b4-0a819bb3ce49 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:28.703682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedf77e2-ff26-49ab-80b4-0a819bb3ce49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:28.703948 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedf77e2-ff26-49ab-80b4-0a819bb3ce49 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:28.708127 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dedf77e2-ff26-49ab-80b4-0a819bb3ce49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:28.708127 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dedf77e2-ff26-49ab-80b4-0a819bb3ce49 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:28.717330 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1205/2421] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:07:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:07:28.721314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416443df-8d91-4164-8482-64c7fc62874d admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:28.732015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416443df-8d91-4164-8482-64c7fc62874d admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:28.736655 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6697c64b-38ba-431d-9660-2fc1049452a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:28.737242 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6697c64b-38ba-431d-9660-2fc1049452a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:28.737518 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6697c64b-38ba-431d-9660-2fc1049452a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:28.737789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6697c64b-38ba-431d-9660-2fc1049452a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:28.738492 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6697c64b-38ba-431d-9660-2fc1049452a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:28.742202 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6697c64b-38ba-431d-9660-2fc1049452a1 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:28.750588 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1217/2422] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:07:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:07:28.754598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedf77e2-ff26-49ab-80b4-0a819bb3ce49 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:28.764500 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedf77e2-ff26-49ab-80b4-0a819bb3ce49 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:28.769100 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b603350-318f-445d-bdfd-5d87652c9302 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:28.769724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b603350-318f-445d-bdfd-5d87652c9302 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:28.770005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b603350-318f-445d-bdfd-5d87652c9302 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:28.770421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b603350-318f-445d-bdfd-5d87652c9302 admin admin] PATH_INFO: `/v3/projects/853d8b91b5094dcb9cd8a9c898d4bb94/users/5251c8463b8349008891ae5a9f698821/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:28.787245 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b603350-318f-445d-bdfd-5d87652c9302 admin admin] RBAC: Authorizing `identity:create_grant(project_id=853d8b91b5094dcb9cd8a9c898d4bb94, user_id=5251c8463b8349008891ae5a9f698821, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:28.790098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b603350-318f-445d-bdfd-5d87652c9302 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:28.859901 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1206/2423] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:07:28 2022] PUT /identity/v3/projects/853d8b91b5094dcb9cd8a9c898d4bb94/users/5251c8463b8349008891ae5a9f698821/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:07:28.871481 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a3fd20-e0ca-4b55-abb8-1cee66bea289 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:28.871815 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a3fd20-e0ca-4b55-abb8-1cee66bea289 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:28.872147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7a3fd20-e0ca-4b55-abb8-1cee66bea289 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:28.905946 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-d7a3fd20-e0ca-4b55-abb8-1cee66bea289 None None] MFA Rules not processed for user `5251c8463b8349008891ae5a9f698821`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:07:28.907211 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d7a3fd20-e0ca-4b55-abb8-1cee66bea289 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:28.956371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:07:28.990194 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1218/2424] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:07:28 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:07:29.002282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b603350-318f-445d-bdfd-5d87652c9302 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:29.006490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8b603350-318f-445d-bdfd-5d87652c9302 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:29.117831 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b603350-318f-445d-bdfd-5d87652c9302 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:29.125202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8547eb0-fb1d-4ba5-810e-947e5e4725f6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:29.125768 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8547eb0-fb1d-4ba5-810e-947e5e4725f6 service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:29.126253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8547eb0-fb1d-4ba5-810e-947e5e4725f6 service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:29.126750 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8547eb0-fb1d-4ba5-810e-947e5e4725f6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:29.137805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8547eb0-fb1d-4ba5-810e-947e5e4725f6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:29.141253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8547eb0-fb1d-4ba5-810e-947e5e4725f6 service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:29.163592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1207/2425] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:07:29 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:07:31.248701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a3fd20-e0ca-4b55-abb8-1cee66bea289 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:31.251942 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d7a3fd20-e0ca-4b55-abb8-1cee66bea289 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:31.337279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7a3fd20-e0ca-4b55-abb8-1cee66bea289 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:31.343389 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197a0e89-1c8f-495c-ad6f-7a20dec416b4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:31.344243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197a0e89-1c8f-495c-ad6f-7a20dec416b4 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:31.345147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197a0e89-1c8f-495c-ad6f-7a20dec416b4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:31.345642 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197a0e89-1c8f-495c-ad6f-7a20dec416b4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:31.348754 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-197a0e89-1c8f-495c-ad6f-7a20dec416b4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:31.364461 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197a0e89-1c8f-495c-ad6f-7a20dec416b4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:31.366157 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197a0e89-1c8f-495c-ad6f-7a20dec416b4 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:31.376828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1219/2426] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:07:31 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:07:38.677160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8547eb0-fb1d-4ba5-810e-947e5e4725f6 service nova] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:38.680041 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d8547eb0-fb1d-4ba5-810e-947e5e4725f6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:38.695847 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8547eb0-fb1d-4ba5-810e-947e5e4725f6 service nova] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:38.702025 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a14d13-b9c9-4fa5-9927-0f3346b3635b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:38.702664 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a14d13-b9c9-4fa5-9927-0f3346b3635b service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:38.702963 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a14d13-b9c9-4fa5-9927-0f3346b3635b service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:38.703232 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a14d13-b9c9-4fa5-9927-0f3346b3635b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:38.705170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-68a14d13-b9c9-4fa5-9927-0f3346b3635b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:38.717397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a14d13-b9c9-4fa5-9927-0f3346b3635b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:38.719051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a14d13-b9c9-4fa5-9927-0f3346b3635b service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:38.730907 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1208/2427] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:07:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:07:38.907759 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197a0e89-1c8f-495c-ad6f-7a20dec416b4 service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:38.983908 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197a0e89-1c8f-495c-ad6f-7a20dec416b4 service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:38.988294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357e144c-d7e0-4fe4-bcad-734f04a8731c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:38.988918 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357e144c-d7e0-4fe4-bcad-734f04a8731c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:38.989383 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357e144c-d7e0-4fe4-bcad-734f04a8731c admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:38.989894 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357e144c-d7e0-4fe4-bcad-734f04a8731c admin admin] PATH_INFO: `/v3/users/5251c8463b8349008891ae5a9f698821` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:38.991081 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357e144c-d7e0-4fe4-bcad-734f04a8731c admin admin] RBAC: Authorizing `identity:delete_user(user_id=5251c8463b8349008891ae5a9f698821)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:38.993151 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357e144c-d7e0-4fe4-bcad-734f04a8731c admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:39.197808 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-357e144c-d7e0-4fe4-bcad-734f04a8731c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5251c8463b8349008891ae5a9f698821'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:39.266183 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-357e144c-d7e0-4fe4-bcad-734f04a8731c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5251c8463b8349008891ae5a9f698821'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:39.287203 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-357e144c-d7e0-4fe4-bcad-734f04a8731c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5251c8463b8349008891ae5a9f698821'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:39.288198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-357e144c-d7e0-4fe4-bcad-734f04a8731c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5251c8463b8349008891ae5a9f698821'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:39.306015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-357e144c-d7e0-4fe4-bcad-734f04a8731c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5251c8463b8349008891ae5a9f698821'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:39.318561 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1220/2428] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:07:38 2022] DELETE /identity/v3/users/5251c8463b8349008891ae5a9f698821 => generated 0 bytes in 412 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:07:39.323406 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ff5285-a8a2-4bb8-9125-4d3b8920c13d None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:39.323746 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ff5285-a8a2-4bb8-9125-4d3b8920c13d None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:39.324032 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ff5285-a8a2-4bb8-9125-4d3b8920c13d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:39.360551 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-83ff5285-a8a2-4bb8-9125-4d3b8920c13d None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:07:39.361869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-83ff5285-a8a2-4bb8-9125-4d3b8920c13d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:39.424829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:07:39.441338 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1209/2429] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:07:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:07:39.451893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357e144c-d7e0-4fe4-bcad-734f04a8731c admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:39.453389 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-357e144c-d7e0-4fe4-bcad-734f04a8731c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:39.464593 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:07:39.520608 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357e144c-d7e0-4fe4-bcad-734f04a8731c admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:39.533140 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300eb5df-fb2e-4178-90b7-52031d20c2c8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:39.533771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300eb5df-fb2e-4178-90b7-52031d20c2c8 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:39.534049 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300eb5df-fb2e-4178-90b7-52031d20c2c8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:39.534348 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300eb5df-fb2e-4178-90b7-52031d20c2c8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:39.550525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300eb5df-fb2e-4178-90b7-52031d20c2c8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:39.552164 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300eb5df-fb2e-4178-90b7-52031d20c2c8 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:39.568327 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1221/2430] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:07:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:07:40.010810 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ff5285-a8a2-4bb8-9125-4d3b8920c13d None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:40.012813 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-83ff5285-a8a2-4bb8-9125-4d3b8920c13d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:40.027106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ff5285-a8a2-4bb8-9125-4d3b8920c13d None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:40.037667 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fd881c-4866-4973-96ea-3c061486e834 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:40.038323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fd881c-4866-4973-96ea-3c061486e834 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:40.043519 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fd881c-4866-4973-96ea-3c061486e834 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:40.049979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fd881c-4866-4973-96ea-3c061486e834 admin admin] PATH_INFO: `/v3/projects/853d8b91b5094dcb9cd8a9c898d4bb94` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:40.051420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79fd881c-4866-4973-96ea-3c061486e834 admin admin] RBAC: Authorizing `identity:delete_project(project_id=853d8b91b5094dcb9cd8a9c898d4bb94)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:40.053394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79fd881c-4866-4973-96ea-3c061486e834 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:40.161646 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-79fd881c-4866-4973-96ea-3c061486e834 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '853d8b91b5094dcb9cd8a9c898d4bb94', 'request_id': 'req-79fd881c-4866-4973-96ea-3c061486e834'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:40.277190 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-79fd881c-4866-4973-96ea-3c061486e834 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '853d8b91b5094dcb9cd8a9c898d4bb94', 'request_id': 'req-79fd881c-4866-4973-96ea-3c061486e834'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:40.288196 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-79fd881c-4866-4973-96ea-3c061486e834 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '853d8b91b5094dcb9cd8a9c898d4bb94', 'request_id': 'req-79fd881c-4866-4973-96ea-3c061486e834'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:40.296808 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-79fd881c-4866-4973-96ea-3c061486e834 admin admin] The token cache is being invalidate because project 853d8b91b5094dcb9cd8a9c898d4bb94 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:07:40.297096 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-79fd881c-4866-4973-96ea-3c061486e834 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:40.297746 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-79fd881c-4866-4973-96ea-3c061486e834 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:40.299626 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1210/2431] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:07:40 2022] DELETE /identity/v3/projects/853d8b91b5094dcb9cd8a9c898d4bb94 => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:07:40.304555 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce74f95c-435a-4323-b2fb-98711c475914 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:40.305266 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce74f95c-435a-4323-b2fb-98711c475914 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:40.305357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce74f95c-435a-4323-b2fb-98711c475914 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:40.325152 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-ce74f95c-435a-4323-b2fb-98711c475914 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:07:40.332280 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-ce74f95c-435a-4323-b2fb-98711c475914 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:40.388035 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1222/2432] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:07:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:07:40.396568 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678c470f-a924-4c7a-a41a-b56eb36dd42a None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:40.396861 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678c470f-a924-4c7a-a41a-b56eb36dd42a None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:40.397131 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-678c470f-a924-4c7a-a41a-b56eb36dd42a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:40.417917 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-678c470f-a924-4c7a-a41a-b56eb36dd42a None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:07:40.419084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-678c470f-a924-4c7a-a41a-b56eb36dd42a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:40.427881 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1211/2433] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:07:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:07:40.432803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce74f95c-435a-4323-b2fb-98711c475914 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:40.443117 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce74f95c-435a-4323-b2fb-98711c475914 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:40.448428 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56388b25-5067-419d-a12e-ab433432b6a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:40.449034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56388b25-5067-419d-a12e-ab433432b6a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:40.449317 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56388b25-5067-419d-a12e-ab433432b6a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:40.449591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56388b25-5067-419d-a12e-ab433432b6a4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:40.450246 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56388b25-5067-419d-a12e-ab433432b6a4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:07:40.450603 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56388b25-5067-419d-a12e-ab433432b6a4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:40.452780 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56388b25-5067-419d-a12e-ab433432b6a4 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:40.464918 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1223/2434] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:07:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 14:07:40.470283 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678c470f-a924-4c7a-a41a-b56eb36dd42a None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:40.479593 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-678c470f-a924-4c7a-a41a-b56eb36dd42a None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:40.484895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e274f869-b499-41c7-93d8-35324e27f9b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:40.485507 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e274f869-b499-41c7-93d8-35324e27f9b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:40.485790 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e274f869-b499-41c7-93d8-35324e27f9b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:40.486060 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e274f869-b499-41c7-93d8-35324e27f9b5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:40.486744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e274f869-b499-41c7-93d8-35324e27f9b5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:40.488661 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e274f869-b499-41c7-93d8-35324e27f9b5 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:40.621498 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1212/2435] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:07:40 2022] POST /identity/v3/projects => generated 373 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:07:40.629371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56388b25-5067-419d-a12e-ab433432b6a4 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:40.640010 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56388b25-5067-419d-a12e-ab433432b6a4 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:40.704668 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998faa8c-2b7c-40c0-8f0d-4c711b5b5ee7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:40.705296 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998faa8c-2b7c-40c0-8f0d-4c711b5b5ee7 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:40.705580 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998faa8c-2b7c-40c0-8f0d-4c711b5b5ee7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:40.705858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998faa8c-2b7c-40c0-8f0d-4c711b5b5ee7 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:40.706548 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998faa8c-2b7c-40c0-8f0d-4c711b5b5ee7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:40.708801 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-998faa8c-2b7c-40c0-8f0d-4c711b5b5ee7 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:40.813943 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1224/2436] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:07:40 2022] POST /identity/v3/users => generated 287 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:07:40.818339 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e274f869-b499-41c7-93d8-35324e27f9b5 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:40.829241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e274f869-b499-41c7-93d8-35324e27f9b5 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:40.833841 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732b0a06-bbec-46da-8f59-0082d7139e90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:40.834546 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732b0a06-bbec-46da-8f59-0082d7139e90 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:40.839113 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732b0a06-bbec-46da-8f59-0082d7139e90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:40.839113 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732b0a06-bbec-46da-8f59-0082d7139e90 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:40.839113 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732b0a06-bbec-46da-8f59-0082d7139e90 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:40.839113 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732b0a06-bbec-46da-8f59-0082d7139e90 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:40.850567 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1213/2437] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:07:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:07:40.851994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998faa8c-2b7c-40c0-8f0d-4c711b5b5ee7 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:40.862253 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998faa8c-2b7c-40c0-8f0d-4c711b5b5ee7 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:40.867493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793d0ded-c2ee-4a74-9fb9-ea9bd8707aa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:40.867493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793d0ded-c2ee-4a74-9fb9-ea9bd8707aa4 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:40.867858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793d0ded-c2ee-4a74-9fb9-ea9bd8707aa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:40.868032 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793d0ded-c2ee-4a74-9fb9-ea9bd8707aa4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:40.868678 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793d0ded-c2ee-4a74-9fb9-ea9bd8707aa4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:40.870279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793d0ded-c2ee-4a74-9fb9-ea9bd8707aa4 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:40.881769 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1225/2438] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:07:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:07:40.885968 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732b0a06-bbec-46da-8f59-0082d7139e90 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:40.897721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732b0a06-bbec-46da-8f59-0082d7139e90 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:40.903074 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f871d4-d466-4e8b-ad28-376163a72310 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:40.903673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f871d4-d466-4e8b-ad28-376163a72310 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:40.903948 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f871d4-d466-4e8b-ad28-376163a72310 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:40.904215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f871d4-d466-4e8b-ad28-376163a72310 admin admin] PATH_INFO: `/v3/projects/cde9af51563941bfb0e4e77f411c833d/users/39abe550c3924d76bfa61305e6a61de0/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:40.930882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f871d4-d466-4e8b-ad28-376163a72310 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cde9af51563941bfb0e4e77f411c833d, user_id=39abe550c3924d76bfa61305e6a61de0, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:40.934199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f871d4-d466-4e8b-ad28-376163a72310 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:41.008736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1214/2439] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:07:40 2022] PUT /identity/v3/projects/cde9af51563941bfb0e4e77f411c833d/users/39abe550c3924d76bfa61305e6a61de0/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:07:41.019403 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61d587b-f9ed-45c2-aaf4-d98c66e6656a None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:41.019707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61d587b-f9ed-45c2-aaf4-d98c66e6656a None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:41.019958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61d587b-f9ed-45c2-aaf4-d98c66e6656a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:41.056319 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-f61d587b-f9ed-45c2-aaf4-d98c66e6656a None None] MFA Rules not processed for user `39abe550c3924d76bfa61305e6a61de0`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:07:41.058294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f61d587b-f9ed-45c2-aaf4-d98c66e6656a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:41.078157 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:07:41.149603 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1226/2440] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:07:41 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:07:41.160667 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f871d4-d466-4e8b-ad28-376163a72310 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:41.162526 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-b3f871d4-d466-4e8b-ad28-376163a72310 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:41.255640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f871d4-d466-4e8b-ad28-376163a72310 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:41.262358 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2aa0dc-b55a-4a68-828d-afd1b7903ee6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:41.265410 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2aa0dc-b55a-4a68-828d-afd1b7903ee6 service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:41.265410 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2aa0dc-b55a-4a68-828d-afd1b7903ee6 service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:41.265410 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2aa0dc-b55a-4a68-828d-afd1b7903ee6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:41.271257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:07:41.273780 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2aa0dc-b55a-4a68-828d-afd1b7903ee6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:41.275403 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2aa0dc-b55a-4a68-828d-afd1b7903ee6 service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:41.285713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1215/2441] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:07:41 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:07:43.942534 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb79b88-27ff-4dbf-bc2f-948cc66944e9 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:43.943064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb79b88-27ff-4dbf-bc2f-948cc66944e9 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:43.943178 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb79b88-27ff-4dbf-bc2f-948cc66944e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:43.962876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-4cb79b88-27ff-4dbf-bc2f-948cc66944e9 None None] MFA Rules not processed for user `0db30f990aae49fabf9b8b19cd019de9`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:07:43.964060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-4cb79b88-27ff-4dbf-bc2f-948cc66944e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:44.022166 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1227/2442] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:07:43 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:07:44.513320 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2aa0dc-b55a-4a68-828d-afd1b7903ee6 service nova] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:44.516538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-cb2aa0dc-b55a-4a68-828d-afd1b7903ee6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:44.537730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb79b88-27ff-4dbf-bc2f-948cc66944e9 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:44.593478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2aa0dc-b55a-4a68-828d-afd1b7903ee6 service nova] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:44.597360 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f65985bf-9905-48f0-ba58-7c39f897bf65 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:44.597962 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65985bf-9905-48f0-ba58-7c39f897bf65 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:44.598301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65985bf-9905-48f0-ba58-7c39f897bf65 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:44.598534 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f65985bf-9905-48f0-ba58-7c39f897bf65 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:44.600268 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f65985bf-9905-48f0-ba58-7c39f897bf65 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:44.608097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb79b88-27ff-4dbf-bc2f-948cc66944e9 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:44.611259 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65985bf-9905-48f0-ba58-7c39f897bf65 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:44.612682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f65985bf-9905-48f0-ba58-7c39f897bf65 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:44.613439 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75bbcc7-9ea0-4ee9-ade9-9703a6813d34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:44.614126 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75bbcc7-9ea0-4ee9-ade9-9703a6813d34 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:44.614495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75bbcc7-9ea0-4ee9-ade9-9703a6813d34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:44.614831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75bbcc7-9ea0-4ee9-ade9-9703a6813d34 admin admin] PATH_INFO: `/v3/users/39abe550c3924d76bfa61305e6a61de0` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:44.616011 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75bbcc7-9ea0-4ee9-ade9-9703a6813d34 admin admin] RBAC: Authorizing `identity:delete_user(user_id=39abe550c3924d76bfa61305e6a61de0)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:44.617981 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75bbcc7-9ea0-4ee9-ade9-9703a6813d34 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:44.621878 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1216/2443] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:07:44 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:07:44.828090 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-a75bbcc7-9ea0-4ee9-ade9-9703a6813d34 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '39abe550c3924d76bfa61305e6a61de0'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:44.831836 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:07:44.899416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-a75bbcc7-9ea0-4ee9-ade9-9703a6813d34 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '39abe550c3924d76bfa61305e6a61de0'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:44.916085 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-a75bbcc7-9ea0-4ee9-ade9-9703a6813d34 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '39abe550c3924d76bfa61305e6a61de0'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:44.916934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-a75bbcc7-9ea0-4ee9-ade9-9703a6813d34 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '39abe550c3924d76bfa61305e6a61de0'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:44.926021 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-a75bbcc7-9ea0-4ee9-ade9-9703a6813d34 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '39abe550c3924d76bfa61305e6a61de0'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:44.932424 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1228/2444] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:07:44 2022] DELETE /identity/v3/users/39abe550c3924d76bfa61305e6a61de0 => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:07:44.936918 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2e18a2-5005-47d3-a0eb-ec12c8a1c980 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:44.937199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2e18a2-5005-47d3-a0eb-ec12c8a1c980 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:44.937706 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2e18a2-5005-47d3-a0eb-ec12c8a1c980 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:44.958042 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-4d2e18a2-5005-47d3-a0eb-ec12c8a1c980 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:07:44.959306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-4d2e18a2-5005-47d3-a0eb-ec12c8a1c980 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:45.025584 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1217/2445] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:07:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:07:45.035598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75bbcc7-9ea0-4ee9-ade9-9703a6813d34 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:45.037932 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a75bbcc7-9ea0-4ee9-ade9-9703a6813d34 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:45.096939 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75bbcc7-9ea0-4ee9-ade9-9703a6813d34 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:45.100661 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2019077e-4f19-41f0-a5df-3b41c8177869 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:45.101270 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2019077e-4f19-41f0-a5df-3b41c8177869 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:45.101550 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2019077e-4f19-41f0-a5df-3b41c8177869 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:45.101824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2019077e-4f19-41f0-a5df-3b41c8177869 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:45.111466 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2019077e-4f19-41f0-a5df-3b41c8177869 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:45.112898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2019077e-4f19-41f0-a5df-3b41c8177869 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:45.123432 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1229/2446] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:07:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:07:45.610701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2e18a2-5005-47d3-a0eb-ec12c8a1c980 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:45.612267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-4d2e18a2-5005-47d3-a0eb-ec12c8a1c980 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:45.634736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2e18a2-5005-47d3-a0eb-ec12c8a1c980 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:45.643227 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7217d497-c0bc-4e03-9811-50de6e78f90f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:45.644025 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7217d497-c0bc-4e03-9811-50de6e78f90f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:45.644546 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7217d497-c0bc-4e03-9811-50de6e78f90f admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:45.645084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7217d497-c0bc-4e03-9811-50de6e78f90f admin admin] PATH_INFO: `/v3/projects/cde9af51563941bfb0e4e77f411c833d` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:45.647298 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7217d497-c0bc-4e03-9811-50de6e78f90f admin admin] RBAC: Authorizing `identity:delete_project(project_id=cde9af51563941bfb0e4e77f411c833d)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:45.649600 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7217d497-c0bc-4e03-9811-50de6e78f90f admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:45.777576 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-7217d497-c0bc-4e03-9811-50de6e78f90f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cde9af51563941bfb0e4e77f411c833d', 'request_id': 'req-7217d497-c0bc-4e03-9811-50de6e78f90f'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:45.848922 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2019077e-4f19-41f0-a5df-3b41c8177869 service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:45.850366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-2019077e-4f19-41f0-a5df-3b41c8177869 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:45.864704 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-7217d497-c0bc-4e03-9811-50de6e78f90f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cde9af51563941bfb0e4e77f411c833d', 'request_id': 'req-7217d497-c0bc-4e03-9811-50de6e78f90f'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:45.872566 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-7217d497-c0bc-4e03-9811-50de6e78f90f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cde9af51563941bfb0e4e77f411c833d', 'request_id': 'req-7217d497-c0bc-4e03-9811-50de6e78f90f'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:45.884998 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-7217d497-c0bc-4e03-9811-50de6e78f90f admin admin] The token cache is being invalidate because project cde9af51563941bfb0e4e77f411c833d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:07:45.885381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-7217d497-c0bc-4e03-9811-50de6e78f90f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:45.886182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-7217d497-c0bc-4e03-9811-50de6e78f90f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:07:45.888010 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1218/2447] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:07:45 2022] DELETE /identity/v3/projects/cde9af51563941bfb0e4e77f411c833d => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:07:45.893185 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdb6bfa-1565-451c-84bf-33d753b1ba43 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:45.893474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdb6bfa-1565-451c-84bf-33d753b1ba43 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:45.893752 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cdb6bfa-1565-451c-84bf-33d753b1ba43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:45.923172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-2cdb6bfa-1565-451c-84bf-33d753b1ba43 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:07:45.924407 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2cdb6bfa-1565-451c-84bf-33d753b1ba43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:45.941634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2019077e-4f19-41f0-a5df-3b41c8177869 service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:45.944940 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:07:45.998407 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1219/2448] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:07:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:07:46.008179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cfc43b-9add-4777-b5a2-453a81077224 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:46.008584 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cfc43b-9add-4777-b5a2-453a81077224 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:46.009256 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cfc43b-9add-4777-b5a2-453a81077224 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:46.019324 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd102f7-2314-4a0f-9d2d-e31394a7561a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:46.019938 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd102f7-2314-4a0f-9d2d-e31394a7561a service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:46.020223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd102f7-2314-4a0f-9d2d-e31394a7561a service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:46.020494 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd102f7-2314-4a0f-9d2d-e31394a7561a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:46.022402 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-9fd102f7-2314-4a0f-9d2d-e31394a7561a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:46.031573 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-92cfc43b-9add-4777-b5a2-453a81077224 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:07:46.033188 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-92cfc43b-9add-4777-b5a2-453a81077224 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:46.035148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:07:46.037708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd102f7-2314-4a0f-9d2d-e31394a7561a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:46.039368 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd102f7-2314-4a0f-9d2d-e31394a7561a service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:46.045608 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1220/2449] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:07:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:07:46.048052 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cfc43b-9add-4777-b5a2-453a81077224 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:46.051762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1230/2450] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:07:45 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:07:46.058531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cfc43b-9add-4777-b5a2-453a81077224 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:46.065467 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb2e990-b473-4b80-badd-2422466af8fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:46.066240 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb2e990-b473-4b80-badd-2422466af8fb admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:46.066671 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb2e990-b473-4b80-badd-2422466af8fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:46.066944 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb2e990-b473-4b80-badd-2422466af8fb admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:46.067599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb2e990-b473-4b80-badd-2422466af8fb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:07:46.067907 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb2e990-b473-4b80-badd-2422466af8fb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:46.069492 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb2e990-b473-4b80-badd-2422466af8fb admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:46.081744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1221/2451] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:07:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:07:46.085977 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd102f7-2314-4a0f-9d2d-e31394a7561a service nova] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:46.102777 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd102f7-2314-4a0f-9d2d-e31394a7561a service nova] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:46.107788 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547179e5-671e-4201-9d0c-912e8d599dd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:46.108399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547179e5-671e-4201-9d0c-912e8d599dd6 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:46.108705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547179e5-671e-4201-9d0c-912e8d599dd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:46.109591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547179e5-671e-4201-9d0c-912e8d599dd6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:46.110351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547179e5-671e-4201-9d0c-912e8d599dd6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:46.112332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547179e5-671e-4201-9d0c-912e8d599dd6 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:46.259789 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1231/2452] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:07:46 2022] POST /identity/v3/projects => generated 369 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:07:46.260740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb2e990-b473-4b80-badd-2422466af8fb admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:46.274282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb2e990-b473-4b80-badd-2422466af8fb admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:46.350790 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b461a7-7f32-4782-9196-7d6000af97e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:46.351483 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b461a7-7f32-4782-9196-7d6000af97e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:46.352231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b461a7-7f32-4782-9196-7d6000af97e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:46.352231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b461a7-7f32-4782-9196-7d6000af97e1 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:46.352680 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b461a7-7f32-4782-9196-7d6000af97e1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:46.354719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b461a7-7f32-4782-9196-7d6000af97e1 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:46.428045 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1222/2453] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:07:46 2022] POST /identity/v3/users => generated 285 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:07:46.432611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547179e5-671e-4201-9d0c-912e8d599dd6 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:46.447913 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547179e5-671e-4201-9d0c-912e8d599dd6 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:46.453310 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da32b1c-eee0-4575-ad70-276a9bfae34a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:46.454130 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da32b1c-eee0-4575-ad70-276a9bfae34a admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:46.454530 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da32b1c-eee0-4575-ad70-276a9bfae34a admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:46.454927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da32b1c-eee0-4575-ad70-276a9bfae34a admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:46.455726 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da32b1c-eee0-4575-ad70-276a9bfae34a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:46.457570 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da32b1c-eee0-4575-ad70-276a9bfae34a admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:46.470388 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1232/2454] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:07:46 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:07:46.473862 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b461a7-7f32-4782-9196-7d6000af97e1 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:46.488117 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b461a7-7f32-4782-9196-7d6000af97e1 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:46.493307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b762028-2f44-40dd-bd87-01b3249fc8dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:46.494034 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b762028-2f44-40dd-bd87-01b3249fc8dd admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:46.494318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b762028-2f44-40dd-bd87-01b3249fc8dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:46.494632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b762028-2f44-40dd-bd87-01b3249fc8dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:46.496095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b762028-2f44-40dd-bd87-01b3249fc8dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:46.498662 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b762028-2f44-40dd-bd87-01b3249fc8dd admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:46.518504 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1223/2455] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:07:46 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:07:46.522088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da32b1c-eee0-4575-ad70-276a9bfae34a admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:46.546821 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da32b1c-eee0-4575-ad70-276a9bfae34a admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:46.552655 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0781d186-4737-42c9-ab38-b38295156289 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:46.553333 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0781d186-4737-42c9-ab38-b38295156289 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:46.553617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0781d186-4737-42c9-ab38-b38295156289 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:46.553894 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0781d186-4737-42c9-ab38-b38295156289 admin admin] PATH_INFO: `/v3/projects/40d8fe457cfa4433aa4a53d1d133ba11/users/afa70c66533a41b79f4b04be1ad323bd/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:46.573001 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0781d186-4737-42c9-ab38-b38295156289 admin admin] RBAC: Authorizing `identity:create_grant(project_id=40d8fe457cfa4433aa4a53d1d133ba11, user_id=afa70c66533a41b79f4b04be1ad323bd, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:46.575777 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0781d186-4737-42c9-ab38-b38295156289 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:46.680889 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1233/2456] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:07:46 2022] PUT /identity/v3/projects/40d8fe457cfa4433aa4a53d1d133ba11/users/afa70c66533a41b79f4b04be1ad323bd/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:07:46.690982 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8239cc9-933f-4cb4-9294-2567f0d6a2a4 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:46.691528 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8239cc9-933f-4cb4-9294-2567f0d6a2a4 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:46.691528 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8239cc9-933f-4cb4-9294-2567f0d6a2a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:46.723931 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-c8239cc9-933f-4cb4-9294-2567f0d6a2a4 None None] MFA Rules not processed for user `afa70c66533a41b79f4b04be1ad323bd`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:07:46.725288 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-c8239cc9-933f-4cb4-9294-2567f0d6a2a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:46.794791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1224/2457] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:07:46 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:07:46.799292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0781d186-4737-42c9-ab38-b38295156289 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:46.875841 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0781d186-4737-42c9-ab38-b38295156289 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:46.882790 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5348653c-9760-498b-8dcc-1576c6bbde68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:46.883607 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5348653c-9760-498b-8dcc-1576c6bbde68 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:46.883988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5348653c-9760-498b-8dcc-1576c6bbde68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:46.884361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5348653c-9760-498b-8dcc-1576c6bbde68 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:46.885146 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5348653c-9760-498b-8dcc-1576c6bbde68 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:46.887605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5348653c-9760-498b-8dcc-1576c6bbde68 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:46.994465 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1234/2458] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:07:46 2022] POST /identity/v3/projects => generated 373 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:07:46.998618 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8239cc9-933f-4cb4-9294-2567f0d6a2a4 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:47.009346 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8239cc9-933f-4cb4-9294-2567f0d6a2a4 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:47.017479 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:07:47.065600 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1156b1-4db6-47f6-bdc8-c9ddbbb51308 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:47.066205 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1156b1-4db6-47f6-bdc8-c9ddbbb51308 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:47.066516 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1156b1-4db6-47f6-bdc8-c9ddbbb51308 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:47.066787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1156b1-4db6-47f6-bdc8-c9ddbbb51308 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:47.067421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1156b1-4db6-47f6-bdc8-c9ddbbb51308 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:47.069374 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1156b1-4db6-47f6-bdc8-c9ddbbb51308 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:47.145248 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1225/2459] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:07:46 2022] POST /identity/v3/users => generated 293 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:07:47.149245 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5348653c-9760-498b-8dcc-1576c6bbde68 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:47.159604 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5348653c-9760-498b-8dcc-1576c6bbde68 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:47.164259 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72541e97-d1c1-4f17-ae42-c717e05b0d98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:47.168259 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72541e97-d1c1-4f17-ae42-c717e05b0d98 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:47.168259 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72541e97-d1c1-4f17-ae42-c717e05b0d98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:47.168259 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72541e97-d1c1-4f17-ae42-c717e05b0d98 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:47.168259 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72541e97-d1c1-4f17-ae42-c717e05b0d98 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:47.168259 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72541e97-d1c1-4f17-ae42-c717e05b0d98 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:47.180575 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1235/2460] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:07:47 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:07:47.186192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1156b1-4db6-47f6-bdc8-c9ddbbb51308 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:47.195323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1156b1-4db6-47f6-bdc8-c9ddbbb51308 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:47.200577 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c234dfd-593c-4890-b08b-96bdae961840 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:47.201307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c234dfd-593c-4890-b08b-96bdae961840 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:47.201582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c234dfd-593c-4890-b08b-96bdae961840 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:47.201852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c234dfd-593c-4890-b08b-96bdae961840 admin admin] PATH_INFO: `/v3/projects/bc4c978f05a1420c85e8e8d087735e58/users/9292eff55dfd418fa29301da4348efd8/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:47.217453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c234dfd-593c-4890-b08b-96bdae961840 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bc4c978f05a1420c85e8e8d087735e58, user_id=9292eff55dfd418fa29301da4348efd8, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:47.220226 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c234dfd-593c-4890-b08b-96bdae961840 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:47.276609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1226/2461] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:07:47 2022] PUT /identity/v3/projects/bc4c978f05a1420c85e8e8d087735e58/users/9292eff55dfd418fa29301da4348efd8/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:07:47.281479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72541e97-d1c1-4f17-ae42-c717e05b0d98 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:47.292103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72541e97-d1c1-4f17-ae42-c717e05b0d98 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:47.356603 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde59815-661f-4664-b374-456ac0ae0051 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:47.357660 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde59815-661f-4664-b374-456ac0ae0051 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:47.357941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde59815-661f-4664-b374-456ac0ae0051 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:47.358205 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde59815-661f-4664-b374-456ac0ae0051 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:47.359440 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde59815-661f-4664-b374-456ac0ae0051 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:47.361088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde59815-661f-4664-b374-456ac0ae0051 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:47.371892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1236/2462] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:07:47 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:07:47.376103 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c234dfd-593c-4890-b08b-96bdae961840 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:47.387588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c234dfd-593c-4890-b08b-96bdae961840 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:47.393999 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c539e4-3533-4102-b4aa-955c834e24d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:47.394820 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c539e4-3533-4102-b4aa-955c834e24d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:47.395090 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c539e4-3533-4102-b4aa-955c834e24d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:47.395359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c539e4-3533-4102-b4aa-955c834e24d9 admin admin] PATH_INFO: `/v3/projects/bc4c978f05a1420c85e8e8d087735e58/users/9292eff55dfd418fa29301da4348efd8/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:47.397274 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c539e4-3533-4102-b4aa-955c834e24d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bc4c978f05a1420c85e8e8d087735e58, user_id=9292eff55dfd418fa29301da4348efd8, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:47.399942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c539e4-3533-4102-b4aa-955c834e24d9 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:47.411320 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1227/2463] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:07:47 2022] PUT /identity/v3/projects/bc4c978f05a1420c85e8e8d087735e58/users/9292eff55dfd418fa29301da4348efd8/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:07:47.422604 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003bd6b2-2bf0-4384-b96c-f09c7eee3756 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:47.422965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003bd6b2-2bf0-4384-b96c-f09c7eee3756 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:47.423298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003bd6b2-2bf0-4384-b96c-f09c7eee3756 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:47.443508 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:07:47.452073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-003bd6b2-2bf0-4384-b96c-f09c7eee3756 None None] MFA Rules not processed for user `9292eff55dfd418fa29301da4348efd8`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:07:47.453351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-003bd6b2-2bf0-4384-b96c-f09c7eee3756 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:47.532292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1237/2464] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:07:47 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:07:47.542922 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c539e4-3533-4102-b4aa-955c834e24d9 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:47.544649 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-15c539e4-3533-4102-b4aa-955c834e24d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:47.610578 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c539e4-3533-4102-b4aa-955c834e24d9 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:47.615742 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f726ca-ddfa-4724-8155-3db86f4dc738 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:47.617476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f726ca-ddfa-4724-8155-3db86f4dc738 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:47.617761 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f726ca-ddfa-4724-8155-3db86f4dc738 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:47.618092 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f726ca-ddfa-4724-8155-3db86f4dc738 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:47.628605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f726ca-ddfa-4724-8155-3db86f4dc738 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:47.630044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f726ca-ddfa-4724-8155-3db86f4dc738 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:47.641897 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1228/2465] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:07:47 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:07:47.859376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003bd6b2-2bf0-4384-b96c-f09c7eee3756 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:47.869632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003bd6b2-2bf0-4384-b96c-f09c7eee3756 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:47.877922 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a6878f-d105-4099-8cfe-934736946307 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:47.877922 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a6878f-d105-4099-8cfe-934736946307 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:47.877922 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a6878f-d105-4099-8cfe-934736946307 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:47.877922 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a6878f-d105-4099-8cfe-934736946307 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:47.936094 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a6878f-d105-4099-8cfe-934736946307 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:47.937620 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a6878f-d105-4099-8cfe-934736946307 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:47.946956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1238/2466] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:07:47 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:07:57.218386 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f726ca-ddfa-4724-8155-3db86f4dc738 service cinder] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:07:57.219905 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d5f726ca-ddfa-4724-8155-3db86f4dc738 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:57.286998 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:07:57.291742 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f726ca-ddfa-4724-8155-3db86f4dc738 service cinder] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:07:57.297748 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcad24cd-175e-42ef-b98f-6332bd3546d9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:07:57.298321 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcad24cd-175e-42ef-b98f-6332bd3546d9 service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:07:57.298633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcad24cd-175e-42ef-b98f-6332bd3546d9 service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:07:57.298930 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcad24cd-175e-42ef-b98f-6332bd3546d9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:07:57.300658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-fcad24cd-175e-42ef-b98f-6332bd3546d9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:07:57.312726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcad24cd-175e-42ef-b98f-6332bd3546d9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:07:57.315112 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcad24cd-175e-42ef-b98f-6332bd3546d9 service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:07:57.325904 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1229/2467] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:07:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:07:57.942682 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=58 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:08:07.584422 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=4 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:08:35.496795 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a6878f-d105-4099-8cfe-934736946307 service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:35.559287 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a6878f-d105-4099-8cfe-934736946307 service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:35.563842 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87dbd8c9-5d46-4cce-9a3b-0aa3c2838c36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:35.564596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87dbd8c9-5d46-4cce-9a3b-0aa3c2838c36 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:35.564874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87dbd8c9-5d46-4cce-9a3b-0aa3c2838c36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:35.565144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87dbd8c9-5d46-4cce-9a3b-0aa3c2838c36 admin admin] PATH_INFO: `/v3/users/afa70c66533a41b79f4b04be1ad323bd` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:35.566259 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87dbd8c9-5d46-4cce-9a3b-0aa3c2838c36 admin admin] RBAC: Authorizing `identity:delete_user(user_id=afa70c66533a41b79f4b04be1ad323bd)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:35.568903 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87dbd8c9-5d46-4cce-9a3b-0aa3c2838c36 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:35.770797 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-87dbd8c9-5d46-4cce-9a3b-0aa3c2838c36 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'afa70c66533a41b79f4b04be1ad323bd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:35.859780 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-87dbd8c9-5d46-4cce-9a3b-0aa3c2838c36 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'afa70c66533a41b79f4b04be1ad323bd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:35.874289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-87dbd8c9-5d46-4cce-9a3b-0aa3c2838c36 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'afa70c66533a41b79f4b04be1ad323bd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:35.875006 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-87dbd8c9-5d46-4cce-9a3b-0aa3c2838c36 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'afa70c66533a41b79f4b04be1ad323bd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:35.883152 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-87dbd8c9-5d46-4cce-9a3b-0aa3c2838c36 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'afa70c66533a41b79f4b04be1ad323bd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:35.889229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1239/2468] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:08:35 2022] DELETE /identity/v3/users/afa70c66533a41b79f4b04be1ad323bd => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:08:35.893966 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1501e94-2fd7-486e-b33c-301daf4f248e None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:35.894482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1501e94-2fd7-486e-b33c-301daf4f248e None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:35.894579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1501e94-2fd7-486e-b33c-301daf4f248e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:35.915009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-a1501e94-2fd7-486e-b33c-301daf4f248e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:08:35.916283 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a1501e94-2fd7-486e-b33c-301daf4f248e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:35.976048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1230/2469] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:08:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:08:35.985590 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87dbd8c9-5d46-4cce-9a3b-0aa3c2838c36 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:35.987001 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-87dbd8c9-5d46-4cce-9a3b-0aa3c2838c36 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:36.048106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87dbd8c9-5d46-4cce-9a3b-0aa3c2838c36 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:36.051655 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe744a5-56f4-4c46-bdf5-8fa247a275c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:36.052256 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe744a5-56f4-4c46-bdf5-8fa247a275c7 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:36.053131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe744a5-56f4-4c46-bdf5-8fa247a275c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:36.053614 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe744a5-56f4-4c46-bdf5-8fa247a275c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:36.062905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe744a5-56f4-4c46-bdf5-8fa247a275c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:36.064328 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe744a5-56f4-4c46-bdf5-8fa247a275c7 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:36.074638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1240/2470] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:08:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:08:36.501552 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1501e94-2fd7-486e-b33c-301daf4f248e None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:36.503024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a1501e94-2fd7-486e-b33c-301daf4f248e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:36.517715 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1501e94-2fd7-486e-b33c-301daf4f248e None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:36.523909 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446a30d9-15d3-4174-9eb4-00ce184ccbd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:36.524563 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446a30d9-15d3-4174-9eb4-00ce184ccbd5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:36.524839 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446a30d9-15d3-4174-9eb4-00ce184ccbd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:36.525186 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446a30d9-15d3-4174-9eb4-00ce184ccbd5 admin admin] PATH_INFO: `/v3/projects/40d8fe457cfa4433aa4a53d1d133ba11` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:36.526293 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446a30d9-15d3-4174-9eb4-00ce184ccbd5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=40d8fe457cfa4433aa4a53d1d133ba11)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:36.528113 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446a30d9-15d3-4174-9eb4-00ce184ccbd5 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:36.671251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-446a30d9-15d3-4174-9eb4-00ce184ccbd5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '40d8fe457cfa4433aa4a53d1d133ba11', 'request_id': 'req-446a30d9-15d3-4174-9eb4-00ce184ccbd5'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:36.757385 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-446a30d9-15d3-4174-9eb4-00ce184ccbd5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '40d8fe457cfa4433aa4a53d1d133ba11', 'request_id': 'req-446a30d9-15d3-4174-9eb4-00ce184ccbd5'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:36.763484 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-446a30d9-15d3-4174-9eb4-00ce184ccbd5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '40d8fe457cfa4433aa4a53d1d133ba11', 'request_id': 'req-446a30d9-15d3-4174-9eb4-00ce184ccbd5'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:36.775338 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-446a30d9-15d3-4174-9eb4-00ce184ccbd5 admin admin] The token cache is being invalidate because project 40d8fe457cfa4433aa4a53d1d133ba11 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:08:36.775629 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-446a30d9-15d3-4174-9eb4-00ce184ccbd5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:36.776292 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-446a30d9-15d3-4174-9eb4-00ce184ccbd5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:36.777782 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1231/2471] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:08:36 2022] DELETE /identity/v3/projects/40d8fe457cfa4433aa4a53d1d133ba11 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:08:36.781979 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe744a5-56f4-4c46-bdf5-8fa247a275c7 service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:36.792341 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe744a5-56f4-4c46-bdf5-8fa247a275c7 service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:36.811027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:08:36.848612 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb63a5b-cd8b-43c6-858f-498734b7f60f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:36.849291 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb63a5b-cd8b-43c6-858f-498734b7f60f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:36.849960 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb63a5b-cd8b-43c6-858f-498734b7f60f admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:36.850281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb63a5b-cd8b-43c6-858f-498734b7f60f admin admin] PATH_INFO: `/v3/users/9292eff55dfd418fa29301da4348efd8` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:36.851416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb63a5b-cd8b-43c6-858f-498734b7f60f admin admin] RBAC: Authorizing `identity:delete_user(user_id=9292eff55dfd418fa29301da4348efd8)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:36.853327 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb63a5b-cd8b-43c6-858f-498734b7f60f admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:37.028836 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6cb63a5b-cd8b-43c6-858f-498734b7f60f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9292eff55dfd418fa29301da4348efd8'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:37.106140 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6cb63a5b-cd8b-43c6-858f-498734b7f60f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9292eff55dfd418fa29301da4348efd8'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:37.120073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6cb63a5b-cd8b-43c6-858f-498734b7f60f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9292eff55dfd418fa29301da4348efd8'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:37.120770 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6cb63a5b-cd8b-43c6-858f-498734b7f60f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9292eff55dfd418fa29301da4348efd8'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:37.127885 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6cb63a5b-cd8b-43c6-858f-498734b7f60f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9292eff55dfd418fa29301da4348efd8'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:37.133902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1241/2472] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:08:36 2022] DELETE /identity/v3/users/9292eff55dfd418fa29301da4348efd8 => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:08:37.501914 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446a30d9-15d3-4174-9eb4-00ce184ccbd5 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:37.502999 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-446a30d9-15d3-4174-9eb4-00ce184ccbd5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:37.516128 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446a30d9-15d3-4174-9eb4-00ce184ccbd5 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:37.573650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ad37ac-2df6-44da-b32a-a7b1cb30bd12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:37.574268 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ad37ac-2df6-44da-b32a-a7b1cb30bd12 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:37.574572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ad37ac-2df6-44da-b32a-a7b1cb30bd12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:37.574841 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ad37ac-2df6-44da-b32a-a7b1cb30bd12 admin admin] PATH_INFO: `/v3/projects/bc4c978f05a1420c85e8e8d087735e58` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:37.575951 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ad37ac-2df6-44da-b32a-a7b1cb30bd12 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bc4c978f05a1420c85e8e8d087735e58)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:37.577852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ad37ac-2df6-44da-b32a-a7b1cb30bd12 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:37.661274 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:08:37.677805 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-96ad37ac-2df6-44da-b32a-a7b1cb30bd12 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bc4c978f05a1420c85e8e8d087735e58', 'request_id': 'req-96ad37ac-2df6-44da-b32a-a7b1cb30bd12'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:37.722235 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-96ad37ac-2df6-44da-b32a-a7b1cb30bd12 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bc4c978f05a1420c85e8e8d087735e58', 'request_id': 'req-96ad37ac-2df6-44da-b32a-a7b1cb30bd12'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:37.728597 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-96ad37ac-2df6-44da-b32a-a7b1cb30bd12 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bc4c978f05a1420c85e8e8d087735e58', 'request_id': 'req-96ad37ac-2df6-44da-b32a-a7b1cb30bd12'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:37.739469 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-96ad37ac-2df6-44da-b32a-a7b1cb30bd12 admin admin] The token cache is being invalidate because project bc4c978f05a1420c85e8e8d087735e58 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:08:37.740026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-96ad37ac-2df6-44da-b32a-a7b1cb30bd12 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:37.740835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-96ad37ac-2df6-44da-b32a-a7b1cb30bd12 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:37.742765 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1232/2473] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:08:37 2022] DELETE /identity/v3/projects/bc4c978f05a1420c85e8e8d087735e58 => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:08:37.747766 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b7699c-5432-49b0-bb52-c071cd876f9d None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:37.748035 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b7699c-5432-49b0-bb52-c071cd876f9d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:37.748308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b7699c-5432-49b0-bb52-c071cd876f9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:37.768421 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-32b7699c-5432-49b0-bb52-c071cd876f9d None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:08:37.769630 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-32b7699c-5432-49b0-bb52-c071cd876f9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:37.827945 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1242/2474] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:08:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:08:37.836437 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e73d88e-5d30-42fe-aa92-84e01d8bad2c None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:37.836729 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e73d88e-5d30-42fe-aa92-84e01d8bad2c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:37.837009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e73d88e-5d30-42fe-aa92-84e01d8bad2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:37.858055 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-8e73d88e-5d30-42fe-aa92-84e01d8bad2c None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:08:37.859288 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8e73d88e-5d30-42fe-aa92-84e01d8bad2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:37.867316 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1233/2475] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:08:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:08:37.871368 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b7699c-5432-49b0-bb52-c071cd876f9d None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:37.881502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b7699c-5432-49b0-bb52-c071cd876f9d None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:37.885902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90f29a9-d005-49a8-8961-3ed556738920 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:37.886522 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90f29a9-d005-49a8-8961-3ed556738920 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:37.886803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90f29a9-d005-49a8-8961-3ed556738920 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:37.887076 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90f29a9-d005-49a8-8961-3ed556738920 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:37.887751 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90f29a9-d005-49a8-8961-3ed556738920 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:08:37.888051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90f29a9-d005-49a8-8961-3ed556738920 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:37.889646 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90f29a9-d005-49a8-8961-3ed556738920 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:37.901701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1243/2476] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:08:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 14:08:37.905841 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e73d88e-5d30-42fe-aa92-84e01d8bad2c None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:37.915336 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e73d88e-5d30-42fe-aa92-84e01d8bad2c None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:37.920093 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf83c6d8-44d6-473a-aa3f-4d4d5f39b883 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:37.920690 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf83c6d8-44d6-473a-aa3f-4d4d5f39b883 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:37.920991 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf83c6d8-44d6-473a-aa3f-4d4d5f39b883 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:37.921264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf83c6d8-44d6-473a-aa3f-4d4d5f39b883 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:37.921919 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf83c6d8-44d6-473a-aa3f-4d4d5f39b883 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:37.923843 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf83c6d8-44d6-473a-aa3f-4d4d5f39b883 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:38.015463 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1234/2477] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:08:37 2022] POST /identity/v3/projects => generated 387 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:08:38.019671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90f29a9-d005-49a8-8961-3ed556738920 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:38.030472 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90f29a9-d005-49a8-8961-3ed556738920 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:38.057501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:08:38.088104 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea5152b-2a1d-42a9-88ca-cdb533654862 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:38.088704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea5152b-2a1d-42a9-88ca-cdb533654862 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:38.088976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea5152b-2a1d-42a9-88ca-cdb533654862 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:38.089243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea5152b-2a1d-42a9-88ca-cdb533654862 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:38.089857 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea5152b-2a1d-42a9-88ca-cdb533654862 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:38.091775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea5152b-2a1d-42a9-88ca-cdb533654862 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:38.171979 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1244/2478] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:08:38 2022] POST /identity/v3/users => generated 294 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:08:38.173457 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf83c6d8-44d6-473a-aa3f-4d4d5f39b883 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:38.184694 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf83c6d8-44d6-473a-aa3f-4d4d5f39b883 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:38.189395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf8a6ee-b888-4c15-8dde-5995790ba704 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:38.189999 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf8a6ee-b888-4c15-8dde-5995790ba704 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:38.193852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf8a6ee-b888-4c15-8dde-5995790ba704 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:38.193852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf8a6ee-b888-4c15-8dde-5995790ba704 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:38.193852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf8a6ee-b888-4c15-8dde-5995790ba704 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:38.193852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf8a6ee-b888-4c15-8dde-5995790ba704 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:38.328848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1235/2479] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:08:38 2022] GET /identity/v3/roles => generated 1420 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:08:38.332895 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea5152b-2a1d-42a9-88ca-cdb533654862 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:38.343325 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea5152b-2a1d-42a9-88ca-cdb533654862 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:38.347836 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a8c41d5-f261-44d3-be73-b36ea6b09aa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:38.348423 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8c41d5-f261-44d3-be73-b36ea6b09aa8 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:38.348697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8c41d5-f261-44d3-be73-b36ea6b09aa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:38.348963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a8c41d5-f261-44d3-be73-b36ea6b09aa8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:38.349771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a8c41d5-f261-44d3-be73-b36ea6b09aa8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:38.351491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a8c41d5-f261-44d3-be73-b36ea6b09aa8 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:38.362025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1245/2480] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:08:38 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:08:38.366997 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf8a6ee-b888-4c15-8dde-5995790ba704 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:38.376922 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf8a6ee-b888-4c15-8dde-5995790ba704 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:38.382209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b36104-ebc1-4191-b97d-5fda92ef558a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:38.382877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b36104-ebc1-4191-b97d-5fda92ef558a admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:38.383156 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b36104-ebc1-4191-b97d-5fda92ef558a admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:38.383426 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b36104-ebc1-4191-b97d-5fda92ef558a admin admin] PATH_INFO: `/v3/projects/830ac297c6684cb7aa07d2f3f765d741/users/7c9401328f4841259bd5892f4dbec207/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:38.397682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b36104-ebc1-4191-b97d-5fda92ef558a admin admin] RBAC: Authorizing `identity:create_grant(project_id=830ac297c6684cb7aa07d2f3f765d741, user_id=7c9401328f4841259bd5892f4dbec207, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:38.400352 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b36104-ebc1-4191-b97d-5fda92ef558a admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:38.471520 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1236/2481] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:08:38 2022] PUT /identity/v3/projects/830ac297c6684cb7aa07d2f3f765d741/users/7c9401328f4841259bd5892f4dbec207/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:08:38.483017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95550b47-e106-4d54-b424-3e27f5ce03d3 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:38.483307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95550b47-e106-4d54-b424-3e27f5ce03d3 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:38.483580 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95550b47-e106-4d54-b424-3e27f5ce03d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:38.511808 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-95550b47-e106-4d54-b424-3e27f5ce03d3 None None] MFA Rules not processed for user `7c9401328f4841259bd5892f4dbec207`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:08:38.513012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-95550b47-e106-4d54-b424-3e27f5ce03d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:38.576282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1246/2482] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:08:38 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:08:38.580640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b36104-ebc1-4191-b97d-5fda92ef558a admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:38.641728 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b36104-ebc1-4191-b97d-5fda92ef558a admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:38.646305 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0fc773-2a0f-432e-ab8b-082b9f72ee6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:38.646984 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0fc773-2a0f-432e-ab8b-082b9f72ee6a admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:38.647233 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0fc773-2a0f-432e-ab8b-082b9f72ee6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:38.647495 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0fc773-2a0f-432e-ab8b-082b9f72ee6a admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:38.648112 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0fc773-2a0f-432e-ab8b-082b9f72ee6a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:38.650009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e0fc773-2a0f-432e-ab8b-082b9f72ee6a admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:38.752311 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1237/2483] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:08:38 2022] POST /identity/v3/projects => generated 385 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:08:38.756418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95550b47-e106-4d54-b424-3e27f5ce03d3 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:38.767109 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95550b47-e106-4d54-b424-3e27f5ce03d3 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:38.823069 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd120dd0-6704-45b1-a98d-e1fca21d1c7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:38.823678 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd120dd0-6704-45b1-a98d-e1fca21d1c7b admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:38.823959 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd120dd0-6704-45b1-a98d-e1fca21d1c7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:38.824232 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd120dd0-6704-45b1-a98d-e1fca21d1c7b admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:38.824864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd120dd0-6704-45b1-a98d-e1fca21d1c7b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:38.826875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd120dd0-6704-45b1-a98d-e1fca21d1c7b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:38.900623 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1247/2484] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:08:38 2022] POST /identity/v3/users => generated 293 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:08:38.904798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0fc773-2a0f-432e-ab8b-082b9f72ee6a admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:38.915798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0fc773-2a0f-432e-ab8b-082b9f72ee6a admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:38.920355 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44e77de-369e-43c1-baaa-1d89ebd823cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:38.921000 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44e77de-369e-43c1-baaa-1d89ebd823cd admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:38.921331 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44e77de-369e-43c1-baaa-1d89ebd823cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:38.921607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44e77de-369e-43c1-baaa-1d89ebd823cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:38.922239 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44e77de-369e-43c1-baaa-1d89ebd823cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:38.923833 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d44e77de-369e-43c1-baaa-1d89ebd823cd admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:38.934198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1238/2485] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:08:38 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:08:38.938109 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd120dd0-6704-45b1-a98d-e1fca21d1c7b admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:38.947959 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd120dd0-6704-45b1-a98d-e1fca21d1c7b admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:38.952581 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1029aef1-1341-44f4-83cd-cf126b62336f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:38.953159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1029aef1-1341-44f4-83cd-cf126b62336f admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:38.953431 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1029aef1-1341-44f4-83cd-cf126b62336f admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:38.953696 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1029aef1-1341-44f4-83cd-cf126b62336f admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:38.954355 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1029aef1-1341-44f4-83cd-cf126b62336f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:38.955922 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1029aef1-1341-44f4-83cd-cf126b62336f admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:38.966537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1248/2486] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:08:38 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:08:38.970514 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44e77de-369e-43c1-baaa-1d89ebd823cd admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:38.981420 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44e77de-369e-43c1-baaa-1d89ebd823cd admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:38.985986 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fe984e-5011-4520-a604-46f1c36e97da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:38.986664 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fe984e-5011-4520-a604-46f1c36e97da admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:38.986974 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fe984e-5011-4520-a604-46f1c36e97da admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:38.987249 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fe984e-5011-4520-a604-46f1c36e97da admin admin] PATH_INFO: `/v3/projects/7fff51338cbf434089c2ee873ab4f2c9/users/565f653ceb1e4022abcebbe706d7f95f/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:39.001665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fe984e-5011-4520-a604-46f1c36e97da admin admin] RBAC: Authorizing `identity:create_grant(project_id=7fff51338cbf434089c2ee873ab4f2c9, user_id=565f653ceb1e4022abcebbe706d7f95f, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:39.004345 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fe984e-5011-4520-a604-46f1c36e97da admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:39.066892 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:DELETE=6,keystone:SELECT=89,keystone:INSERT=5 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:08:39.071613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1239/2487] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:08:38 2022] PUT /identity/v3/projects/7fff51338cbf434089c2ee873ab4f2c9/users/565f653ceb1e4022abcebbe706d7f95f/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:08:39.082095 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef346be9-52dc-49e5-bd81-31664768dde9 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:39.082353 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef346be9-52dc-49e5-bd81-31664768dde9 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:39.082643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef346be9-52dc-49e5-bd81-31664768dde9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:39.112366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-ef346be9-52dc-49e5-bd81-31664768dde9 None None] MFA Rules not processed for user `565f653ceb1e4022abcebbe706d7f95f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:08:39.113601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-ef346be9-52dc-49e5-bd81-31664768dde9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:39.117764 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:08:39.177306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1249/2488] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:08:39 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:08:39.182457 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fe984e-5011-4520-a604-46f1c36e97da admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:39.250308 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fe984e-5011-4520-a604-46f1c36e97da admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:39.254919 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fd04df-3fab-468b-b2f5-92465f607f44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:39.256018 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fd04df-3fab-468b-b2f5-92465f607f44 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:39.256403 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fd04df-3fab-468b-b2f5-92465f607f44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:39.256676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fd04df-3fab-468b-b2f5-92465f607f44 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:39.257325 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fd04df-3fab-468b-b2f5-92465f607f44 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:39.259253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fd04df-3fab-468b-b2f5-92465f607f44 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:39.319064 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1240/2489] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:08:39 2022] POST /identity/v3/projects => generated 385 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:08:39.323036 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef346be9-52dc-49e5-bd81-31664768dde9 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:39.334500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef346be9-52dc-49e5-bd81-31664768dde9 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:39.393287 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c155f1c1-b1ef-49e1-aad7-2c9ba4ae746b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:39.394634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c155f1c1-b1ef-49e1-aad7-2c9ba4ae746b admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:39.394914 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c155f1c1-b1ef-49e1-aad7-2c9ba4ae746b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:39.395186 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c155f1c1-b1ef-49e1-aad7-2c9ba4ae746b admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:39.395889 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c155f1c1-b1ef-49e1-aad7-2c9ba4ae746b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:39.398705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c155f1c1-b1ef-49e1-aad7-2c9ba4ae746b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:39.486314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1250/2490] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:08:39 2022] POST /identity/v3/users => generated 299 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:08:39.487531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fd04df-3fab-468b-b2f5-92465f607f44 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:39.499128 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fd04df-3fab-468b-b2f5-92465f607f44 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:39.503909 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af63a15f-950d-423c-8cfc-1edab3ef175e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:39.504537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af63a15f-950d-423c-8cfc-1edab3ef175e admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:39.504807 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af63a15f-950d-423c-8cfc-1edab3ef175e admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:39.505070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af63a15f-950d-423c-8cfc-1edab3ef175e admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:39.506244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af63a15f-950d-423c-8cfc-1edab3ef175e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:39.507833 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af63a15f-950d-423c-8cfc-1edab3ef175e admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:39.518157 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1241/2491] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:08:39 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:08:39.530613 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c155f1c1-b1ef-49e1-aad7-2c9ba4ae746b admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:39.540824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c155f1c1-b1ef-49e1-aad7-2c9ba4ae746b admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:39.545490 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8227e42-60c8-4ea5-9b51-c1ca072f53bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:39.546139 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8227e42-60c8-4ea5-9b51-c1ca072f53bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:39.546449 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8227e42-60c8-4ea5-9b51-c1ca072f53bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:39.546722 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8227e42-60c8-4ea5-9b51-c1ca072f53bb admin admin] PATH_INFO: `/v3/projects/8a0037efb39a4fa4898c3756e0a00539/users/d6659ee6fe2044f09ebee4c6813a45de/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:39.561910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8227e42-60c8-4ea5-9b51-c1ca072f53bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=8a0037efb39a4fa4898c3756e0a00539, user_id=d6659ee6fe2044f09ebee4c6813a45de, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:39.564621 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8227e42-60c8-4ea5-9b51-c1ca072f53bb admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:39.625604 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1251/2492] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:08:39 2022] PUT /identity/v3/projects/8a0037efb39a4fa4898c3756e0a00539/users/d6659ee6fe2044f09ebee4c6813a45de/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:08:39.629642 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af63a15f-950d-423c-8cfc-1edab3ef175e admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:39.640490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af63a15f-950d-423c-8cfc-1edab3ef175e admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:39.698131 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f199d0-0bdb-4267-a4fa-015e01f13d09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:39.698797 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f199d0-0bdb-4267-a4fa-015e01f13d09 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:39.699074 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f199d0-0bdb-4267-a4fa-015e01f13d09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:39.699345 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f199d0-0bdb-4267-a4fa-015e01f13d09 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:39.699982 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f199d0-0bdb-4267-a4fa-015e01f13d09 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:39.701553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f199d0-0bdb-4267-a4fa-015e01f13d09 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:39.711158 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1242/2493] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:08:39 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:08:39.715334 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8227e42-60c8-4ea5-9b51-c1ca072f53bb admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:39.725583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8227e42-60c8-4ea5-9b51-c1ca072f53bb admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:39.730188 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5508a6e9-24aa-46ed-b1a0-bc17c47bee36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:39.730866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5508a6e9-24aa-46ed-b1a0-bc17c47bee36 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:39.731143 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5508a6e9-24aa-46ed-b1a0-bc17c47bee36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:39.731415 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5508a6e9-24aa-46ed-b1a0-bc17c47bee36 admin admin] PATH_INFO: `/v3/projects/8a0037efb39a4fa4898c3756e0a00539/users/d6659ee6fe2044f09ebee4c6813a45de/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:39.733321 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5508a6e9-24aa-46ed-b1a0-bc17c47bee36 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8a0037efb39a4fa4898c3756e0a00539, user_id=d6659ee6fe2044f09ebee4c6813a45de, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:39.736004 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5508a6e9-24aa-46ed-b1a0-bc17c47bee36 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:39.747514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1252/2494] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:08:39 2022] PUT /identity/v3/projects/8a0037efb39a4fa4898c3756e0a00539/users/d6659ee6fe2044f09ebee4c6813a45de/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:08:39.758297 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec27dd1-35d1-4314-8220-b0388520061d None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:39.758507 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec27dd1-35d1-4314-8220-b0388520061d None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:39.758783 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec27dd1-35d1-4314-8220-b0388520061d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:39.785716 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-cec27dd1-35d1-4314-8220-b0388520061d None None] MFA Rules not processed for user `d6659ee6fe2044f09ebee4c6813a45de`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:08:39.786959 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-cec27dd1-35d1-4314-8220-b0388520061d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:39.845143 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:08:39.854975 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1243/2495] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:08:39 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:08:39.865768 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5508a6e9-24aa-46ed-b1a0-bc17c47bee36 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:39.867506 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-5508a6e9-24aa-46ed-b1a0-bc17c47bee36 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:39.925844 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5508a6e9-24aa-46ed-b1a0-bc17c47bee36 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:39.930190 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04400c7b-d12c-4b41-9c00-90157f68d600 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:39.930810 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04400c7b-d12c-4b41-9c00-90157f68d600 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:39.931087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04400c7b-d12c-4b41-9c00-90157f68d600 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:39.931621 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04400c7b-d12c-4b41-9c00-90157f68d600 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:39.940928 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04400c7b-d12c-4b41-9c00-90157f68d600 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:39.942383 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04400c7b-d12c-4b41-9c00-90157f68d600 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:39.947868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:08:39.952565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1253/2496] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:08:39 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:08:41.073536 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec27dd1-35d1-4314-8220-b0388520061d None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:41.074948 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-cec27dd1-35d1-4314-8220-b0388520061d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:41.086802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec27dd1-35d1-4314-8220-b0388520061d None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:41.090280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2800d73e-f6fd-44cd-aaf0-9957a68cd56b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:41.090900 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2800d73e-f6fd-44cd-aaf0-9957a68cd56b service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:41.091181 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2800d73e-f6fd-44cd-aaf0-9957a68cd56b service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:41.091455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2800d73e-f6fd-44cd-aaf0-9957a68cd56b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:41.150068 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2800d73e-f6fd-44cd-aaf0-9957a68cd56b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:41.151516 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2800d73e-f6fd-44cd-aaf0-9957a68cd56b service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:41.160233 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1244/2497] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:08:41 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:08:46.094858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04400c7b-d12c-4b41-9c00-90157f68d600 service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:46.103508 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04400c7b-d12c-4b41-9c00-90157f68d600 service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:46.109624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f789b09-3e5f-4e1e-9cbc-087c78cdbaa3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:46.109624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f789b09-3e5f-4e1e-9cbc-087c78cdbaa3 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:46.109624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f789b09-3e5f-4e1e-9cbc-087c78cdbaa3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:46.109624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f789b09-3e5f-4e1e-9cbc-087c78cdbaa3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:46.174029 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f789b09-3e5f-4e1e-9cbc-087c78cdbaa3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:46.175546 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f789b09-3e5f-4e1e-9cbc-087c78cdbaa3 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:46.185143 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1254/2498] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:08:46 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:08:46.277774 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c56877-949f-4c6e-9556-d186f55f6ba9 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:46.283167 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c56877-949f-4c6e-9556-d186f55f6ba9 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:46.283167 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c56877-949f-4c6e-9556-d186f55f6ba9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:46.301211 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-30c56877-949f-4c6e-9556-d186f55f6ba9 None None] MFA Rules not processed for user `b471d768cd814ff8b4f6bd35d21e24a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:08:46.302536 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-30c56877-949f-4c6e-9556-d186f55f6ba9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:46.360187 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1245/2499] 10.222.0.62 () {58 vars in 1009 bytes} [Fri Sep 23 14:08:46 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:08:46.505376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f789b09-3e5f-4e1e-9cbc-087c78cdbaa3 service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:46.506844 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-2f789b09-3e5f-4e1e-9cbc-087c78cdbaa3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:46.583953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f789b09-3e5f-4e1e-9cbc-087c78cdbaa3 service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:46.589137 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8ea1c6-081d-42d3-822c-fa801f5ce51c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b9269509f1d454eab7a3008c2ba0039', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:46.589780 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8ea1c6-081d-42d3-822c-fa801f5ce51c service swift] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:46.590057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8ea1c6-081d-42d3-822c-fa801f5ce51c service swift] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:46.590357 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8ea1c6-081d-42d3-822c-fa801f5ce51c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:46.598526 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8ea1c6-081d-42d3-822c-fa801f5ce51c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:46.600094 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8ea1c6-081d-42d3-822c-fa801f5ce51c service swift] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:46.608106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1255/2500] 10.222.0.62 () {60 vars in 1384 bytes} [Fri Sep 23 14:08:46 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 23 14:08:48.303118 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c56877-949f-4c6e-9556-d186f55f6ba9 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:48.305167 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-30c56877-949f-4c6e-9556-d186f55f6ba9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:48.403366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c56877-949f-4c6e-9556-d186f55f6ba9 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:48.409719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4154e48a-55a6-41d1-a845-9c24dbf06415 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:48.410354 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4154e48a-55a6-41d1-a845-9c24dbf06415 service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:48.410639 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4154e48a-55a6-41d1-a845-9c24dbf06415 service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:48.410913 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4154e48a-55a6-41d1-a845-9c24dbf06415 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:48.412917 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-4154e48a-55a6-41d1-a845-9c24dbf06415 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:48.455748 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:08:48.484078 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4154e48a-55a6-41d1-a845-9c24dbf06415 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:48.486091 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4154e48a-55a6-41d1-a845-9c24dbf06415 service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:48.498248 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1246/2501] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:08:48 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:08:50.969347 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8ea1c6-081d-42d3-822c-fa801f5ce51c service swift] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:51.098844 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8ea1c6-081d-42d3-822c-fa801f5ce51c service swift] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:51.108440 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dd3540-b7ee-4ec5-90f8-10da8093f485 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:51.109711 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dd3540-b7ee-4ec5-90f8-10da8093f485 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:51.110665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dd3540-b7ee-4ec5-90f8-10da8093f485 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:51.110942 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8dd3540-b7ee-4ec5-90f8-10da8093f485 admin admin] PATH_INFO: `/v3/users/7c9401328f4841259bd5892f4dbec207` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:51.112106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8dd3540-b7ee-4ec5-90f8-10da8093f485 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7c9401328f4841259bd5892f4dbec207)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:51.114619 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8dd3540-b7ee-4ec5-90f8-10da8093f485 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:51.302110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:08:51.327695 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e8dd3540-b7ee-4ec5-90f8-10da8093f485 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7c9401328f4841259bd5892f4dbec207'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:51.399715 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e8dd3540-b7ee-4ec5-90f8-10da8093f485 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7c9401328f4841259bd5892f4dbec207'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:51.415916 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e8dd3540-b7ee-4ec5-90f8-10da8093f485 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7c9401328f4841259bd5892f4dbec207'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:51.417071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e8dd3540-b7ee-4ec5-90f8-10da8093f485 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7c9401328f4841259bd5892f4dbec207'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:51.430103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-e8dd3540-b7ee-4ec5-90f8-10da8093f485 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7c9401328f4841259bd5892f4dbec207'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:51.440428 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1256/2502] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:08:50 2022] DELETE /identity/v3/users/7c9401328f4841259bd5892f4dbec207 => generated 0 bytes in 471 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:08:51.449766 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a8d83b-e7ea-4491-ab8c-8d62043dc97c None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:51.449766 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a8d83b-e7ea-4491-ab8c-8d62043dc97c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:51.451054 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a8d83b-e7ea-4491-ab8c-8d62043dc97c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:51.490364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-23a8d83b-e7ea-4491-ab8c-8d62043dc97c None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:08:51.491572 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-23a8d83b-e7ea-4491-ab8c-8d62043dc97c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:51.579163 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1247/2503] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:08:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:08:51.588778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dd3540-b7ee-4ec5-90f8-10da8093f485 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:51.590481 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-e8dd3540-b7ee-4ec5-90f8-10da8093f485 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:51.672552 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8dd3540-b7ee-4ec5-90f8-10da8093f485 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:51.675898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254aae14-571b-48bd-828b-0e640bd63d9d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:51.676500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254aae14-571b-48bd-828b-0e640bd63d9d service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:51.676776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254aae14-571b-48bd-828b-0e640bd63d9d service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:51.677047 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254aae14-571b-48bd-828b-0e640bd63d9d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:51.687821 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254aae14-571b-48bd-828b-0e640bd63d9d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:51.689394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254aae14-571b-48bd-828b-0e640bd63d9d service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:51.701538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1257/2504] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:08:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:08:52.254094 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a8d83b-e7ea-4491-ab8c-8d62043dc97c None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:52.255734 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-23a8d83b-e7ea-4491-ab8c-8d62043dc97c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:52.281952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a8d83b-e7ea-4491-ab8c-8d62043dc97c None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:52.289291 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c750a04-1b18-4d50-9c87-1c08a9aeddd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:52.289968 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c750a04-1b18-4d50-9c87-1c08a9aeddd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:52.290365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c750a04-1b18-4d50-9c87-1c08a9aeddd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:52.290653 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c750a04-1b18-4d50-9c87-1c08a9aeddd6 admin admin] PATH_INFO: `/v3/projects/830ac297c6684cb7aa07d2f3f765d741` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:52.299580 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c750a04-1b18-4d50-9c87-1c08a9aeddd6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=830ac297c6684cb7aa07d2f3f765d741)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:52.301642 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c750a04-1b18-4d50-9c87-1c08a9aeddd6 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:52.419413 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1c750a04-1b18-4d50-9c87-1c08a9aeddd6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '830ac297c6684cb7aa07d2f3f765d741', 'request_id': 'req-1c750a04-1b18-4d50-9c87-1c08a9aeddd6'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:52.472068 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1c750a04-1b18-4d50-9c87-1c08a9aeddd6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '830ac297c6684cb7aa07d2f3f765d741', 'request_id': 'req-1c750a04-1b18-4d50-9c87-1c08a9aeddd6'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:52.478670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1c750a04-1b18-4d50-9c87-1c08a9aeddd6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '830ac297c6684cb7aa07d2f3f765d741', 'request_id': 'req-1c750a04-1b18-4d50-9c87-1c08a9aeddd6'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:52.491385 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1c750a04-1b18-4d50-9c87-1c08a9aeddd6 admin admin] The token cache is being invalidate because project 830ac297c6684cb7aa07d2f3f765d741 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:08:52.491789 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1c750a04-1b18-4d50-9c87-1c08a9aeddd6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:52.492466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1c750a04-1b18-4d50-9c87-1c08a9aeddd6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:52.494259 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1248/2505] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:08:52 2022] DELETE /identity/v3/projects/830ac297c6684cb7aa07d2f3f765d741 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:08:52.498498 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254aae14-571b-48bd-828b-0e640bd63d9d service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:52.510177 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254aae14-571b-48bd-828b-0e640bd63d9d service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:52.622027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891139cb-270e-4641-843f-f0d3ca31efb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:52.622855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891139cb-270e-4641-843f-f0d3ca31efb4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:52.623201 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891139cb-270e-4641-843f-f0d3ca31efb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:52.623531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891139cb-270e-4641-843f-f0d3ca31efb4 admin admin] PATH_INFO: `/v3/users/565f653ceb1e4022abcebbe706d7f95f` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:52.624651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891139cb-270e-4641-843f-f0d3ca31efb4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=565f653ceb1e4022abcebbe706d7f95f)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:52.626775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891139cb-270e-4641-843f-f0d3ca31efb4 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:52.802983 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-891139cb-270e-4641-843f-f0d3ca31efb4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '565f653ceb1e4022abcebbe706d7f95f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:52.808876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:08:52.946379 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-891139cb-270e-4641-843f-f0d3ca31efb4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '565f653ceb1e4022abcebbe706d7f95f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:52.961280 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-891139cb-270e-4641-843f-f0d3ca31efb4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '565f653ceb1e4022abcebbe706d7f95f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:52.962057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-891139cb-270e-4641-843f-f0d3ca31efb4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '565f653ceb1e4022abcebbe706d7f95f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:52.970076 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-891139cb-270e-4641-843f-f0d3ca31efb4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '565f653ceb1e4022abcebbe706d7f95f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:52.976621 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1258/2506] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:08:52 2022] DELETE /identity/v3/users/565f653ceb1e4022abcebbe706d7f95f => generated 0 bytes in 479 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:08:53.736843 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c750a04-1b18-4d50-9c87-1c08a9aeddd6 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:53.739197 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-1c750a04-1b18-4d50-9c87-1c08a9aeddd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:53.774107 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c750a04-1b18-4d50-9c87-1c08a9aeddd6 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:53.845225 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516b28aa-616b-4fd1-b5b9-5442768e5b16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:53.846122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516b28aa-616b-4fd1-b5b9-5442768e5b16 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:53.846431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516b28aa-616b-4fd1-b5b9-5442768e5b16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:53.846821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516b28aa-616b-4fd1-b5b9-5442768e5b16 admin admin] PATH_INFO: `/v3/projects/7fff51338cbf434089c2ee873ab4f2c9` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:53.847958 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516b28aa-616b-4fd1-b5b9-5442768e5b16 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7fff51338cbf434089c2ee873ab4f2c9)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:53.850052 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516b28aa-616b-4fd1-b5b9-5442768e5b16 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:53.852515 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:08:53.999350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-516b28aa-616b-4fd1-b5b9-5442768e5b16 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7fff51338cbf434089c2ee873ab4f2c9', 'request_id': 'req-516b28aa-616b-4fd1-b5b9-5442768e5b16'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:54.085121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-516b28aa-616b-4fd1-b5b9-5442768e5b16 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7fff51338cbf434089c2ee873ab4f2c9', 'request_id': 'req-516b28aa-616b-4fd1-b5b9-5442768e5b16'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:54.099733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-516b28aa-616b-4fd1-b5b9-5442768e5b16 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7fff51338cbf434089c2ee873ab4f2c9', 'request_id': 'req-516b28aa-616b-4fd1-b5b9-5442768e5b16'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:54.114116 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-516b28aa-616b-4fd1-b5b9-5442768e5b16 admin admin] The token cache is being invalidate because project 7fff51338cbf434089c2ee873ab4f2c9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:08:54.114534 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-516b28aa-616b-4fd1-b5b9-5442768e5b16 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:54.115890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-516b28aa-616b-4fd1-b5b9-5442768e5b16 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:54.117910 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1249/2507] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:08:53 2022] DELETE /identity/v3/projects/7fff51338cbf434089c2ee873ab4f2c9 => generated 0 bytes in 382 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:08:54.122754 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891139cb-270e-4641-843f-f0d3ca31efb4 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:54.124756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-891139cb-270e-4641-843f-f0d3ca31efb4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:54.138421 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891139cb-270e-4641-843f-f0d3ca31efb4 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:54.226682 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26956e6a-b58b-490a-a76c-00057d65e25b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:54.227906 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26956e6a-b58b-490a-a76c-00057d65e25b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:54.227906 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26956e6a-b58b-490a-a76c-00057d65e25b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:54.227906 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26956e6a-b58b-490a-a76c-00057d65e25b admin admin] PATH_INFO: `/v3/users/d6659ee6fe2044f09ebee4c6813a45de` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:54.229364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26956e6a-b58b-490a-a76c-00057d65e25b admin admin] RBAC: Authorizing `identity:delete_user(user_id=d6659ee6fe2044f09ebee4c6813a45de)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:54.231462 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26956e6a-b58b-490a-a76c-00057d65e25b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:54.398866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-26956e6a-b58b-490a-a76c-00057d65e25b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd6659ee6fe2044f09ebee4c6813a45de'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:54.494903 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-26956e6a-b58b-490a-a76c-00057d65e25b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd6659ee6fe2044f09ebee4c6813a45de'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:54.513838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-26956e6a-b58b-490a-a76c-00057d65e25b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd6659ee6fe2044f09ebee4c6813a45de'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:54.514887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-26956e6a-b58b-490a-a76c-00057d65e25b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd6659ee6fe2044f09ebee4c6813a45de'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:54.528752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-26956e6a-b58b-490a-a76c-00057d65e25b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd6659ee6fe2044f09ebee4c6813a45de'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:54.535574 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1259/2508] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:08:54 2022] DELETE /identity/v3/users/d6659ee6fe2044f09ebee4c6813a45de => generated 0 bytes in 414 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:08:55.137634 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516b28aa-616b-4fd1-b5b9-5442768e5b16 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:55.139419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-516b28aa-616b-4fd1-b5b9-5442768e5b16 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:55.164292 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516b28aa-616b-4fd1-b5b9-5442768e5b16 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:55.248148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f195675-1180-4a5c-b883-b8506e06f6ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:55.249008 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f195675-1180-4a5c-b883-b8506e06f6ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:55.250106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f195675-1180-4a5c-b883-b8506e06f6ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:55.250557 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f195675-1180-4a5c-b883-b8506e06f6ea admin admin] PATH_INFO: `/v3/projects/8a0037efb39a4fa4898c3756e0a00539` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:55.252438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f195675-1180-4a5c-b883-b8506e06f6ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=8a0037efb39a4fa4898c3756e0a00539)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:55.254663 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f195675-1180-4a5c-b883-b8506e06f6ea admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:55.360669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5f195675-1180-4a5c-b883-b8506e06f6ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8a0037efb39a4fa4898c3756e0a00539', 'request_id': 'req-5f195675-1180-4a5c-b883-b8506e06f6ea'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:55.420010 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5f195675-1180-4a5c-b883-b8506e06f6ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8a0037efb39a4fa4898c3756e0a00539', 'request_id': 'req-5f195675-1180-4a5c-b883-b8506e06f6ea'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:55.428421 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5f195675-1180-4a5c-b883-b8506e06f6ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8a0037efb39a4fa4898c3756e0a00539', 'request_id': 'req-5f195675-1180-4a5c-b883-b8506e06f6ea'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:55.453173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5f195675-1180-4a5c-b883-b8506e06f6ea admin admin] The token cache is being invalidate because project 8a0037efb39a4fa4898c3756e0a00539 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:08:55.454445 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5f195675-1180-4a5c-b883-b8506e06f6ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:55.454445 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-5f195675-1180-4a5c-b883-b8506e06f6ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:55.456759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1250/2509] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:08:55 2022] DELETE /identity/v3/projects/8a0037efb39a4fa4898c3756e0a00539 => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:08:55.463065 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a455ee2-e752-4da9-b49d-da87829df7a2 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:55.463585 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a455ee2-e752-4da9-b49d-da87829df7a2 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:55.463585 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a455ee2-e752-4da9-b49d-da87829df7a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:55.491479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-3a455ee2-e752-4da9-b49d-da87829df7a2 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:08:55.498576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-3a455ee2-e752-4da9-b49d-da87829df7a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:55.572338 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:08:55.615287 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1260/2510] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:08:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:08:55.627373 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d150cf2f-6ca5-4408-9db4-00fa5e3e5455 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:55.633181 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d150cf2f-6ca5-4408-9db4-00fa5e3e5455 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:55.633181 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d150cf2f-6ca5-4408-9db4-00fa5e3e5455 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:55.658903 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-d150cf2f-6ca5-4408-9db4-00fa5e3e5455 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:08:55.660330 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d150cf2f-6ca5-4408-9db4-00fa5e3e5455 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:55.673901 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1251/2511] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:08:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:08:55.674530 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a455ee2-e752-4da9-b49d-da87829df7a2 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:55.687855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a455ee2-e752-4da9-b49d-da87829df7a2 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:55.694619 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd56bb85-fddd-49dc-9876-b5aef0985599 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:55.695490 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd56bb85-fddd-49dc-9876-b5aef0985599 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:55.700897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd56bb85-fddd-49dc-9876-b5aef0985599 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:55.700897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd56bb85-fddd-49dc-9876-b5aef0985599 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:55.700897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd56bb85-fddd-49dc-9876-b5aef0985599 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:08:55.700897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd56bb85-fddd-49dc-9876-b5aef0985599 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:55.700897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd56bb85-fddd-49dc-9876-b5aef0985599 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:55.717200 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d150cf2f-6ca5-4408-9db4-00fa5e3e5455 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:55.718521 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1261/2512] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:08:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:08:55.730125 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d150cf2f-6ca5-4408-9db4-00fa5e3e5455 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:55.735032 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a90db2-7318-412d-8e0c-3ebfc9f1a71e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:55.735642 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a90db2-7318-412d-8e0c-3ebfc9f1a71e admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:55.735922 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a90db2-7318-412d-8e0c-3ebfc9f1a71e admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:55.736199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a90db2-7318-412d-8e0c-3ebfc9f1a71e admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:55.736822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a90db2-7318-412d-8e0c-3ebfc9f1a71e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:55.738749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a90db2-7318-412d-8e0c-3ebfc9f1a71e admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:55.751827 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:08:55.878494 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1252/2513] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:08:55 2022] POST /identity/v3/projects => generated 383 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:08:55.883589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd56bb85-fddd-49dc-9876-b5aef0985599 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:55.899554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd56bb85-fddd-49dc-9876-b5aef0985599 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:56.019511 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5398275-d5aa-4439-8daa-082c3f70d0cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:56.021405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5398275-d5aa-4439-8daa-082c3f70d0cf admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:56.022380 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5398275-d5aa-4439-8daa-082c3f70d0cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:56.027159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5398275-d5aa-4439-8daa-082c3f70d0cf admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:56.030754 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5398275-d5aa-4439-8daa-082c3f70d0cf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:56.035203 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5398275-d5aa-4439-8daa-082c3f70d0cf admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:56.168267 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1262/2514] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:08:55 2022] POST /identity/v3/users => generated 292 bytes in 289 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:08:56.169174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a90db2-7318-412d-8e0c-3ebfc9f1a71e admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:56.186200 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a90db2-7318-412d-8e0c-3ebfc9f1a71e admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:56.199147 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac573b4-2a29-42be-a56d-23b60b097840 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:56.200234 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac573b4-2a29-42be-a56d-23b60b097840 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:56.200873 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac573b4-2a29-42be-a56d-23b60b097840 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:56.202268 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac573b4-2a29-42be-a56d-23b60b097840 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:56.203149 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac573b4-2a29-42be-a56d-23b60b097840 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:56.206215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac573b4-2a29-42be-a56d-23b60b097840 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:56.227380 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1253/2515] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:08:56 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:08:56.232117 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5398275-d5aa-4439-8daa-082c3f70d0cf admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:56.248362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5398275-d5aa-4439-8daa-082c3f70d0cf admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:56.253444 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d5a996-5787-47ce-90da-9e8e92487f28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:56.254150 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d5a996-5787-47ce-90da-9e8e92487f28 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:56.254531 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d5a996-5787-47ce-90da-9e8e92487f28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:56.254884 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d5a996-5787-47ce-90da-9e8e92487f28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:56.255597 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d5a996-5787-47ce-90da-9e8e92487f28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:56.257275 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d5a996-5787-47ce-90da-9e8e92487f28 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:56.277692 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac573b4-2a29-42be-a56d-23b60b097840 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:56.278387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1263/2516] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:08:56 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:08:56.289902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac573b4-2a29-42be-a56d-23b60b097840 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:56.296479 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bf9b80-7ace-4288-bc00-a28a1eada848 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:56.297148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bf9b80-7ace-4288-bc00-a28a1eada848 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:56.297658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bf9b80-7ace-4288-bc00-a28a1eada848 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:56.298780 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bf9b80-7ace-4288-bc00-a28a1eada848 admin admin] PATH_INFO: `/v3/projects/ff79a7bbf6b846e5841444acff0cd6d4/users/1ae3afa23f754392af735233b3975b4c/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:56.326927 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bf9b80-7ace-4288-bc00-a28a1eada848 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff79a7bbf6b846e5841444acff0cd6d4, user_id=1ae3afa23f754392af735233b3975b4c, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:56.330198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bf9b80-7ace-4288-bc00-a28a1eada848 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:56.417034 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1254/2517] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:08:56 2022] PUT /identity/v3/projects/ff79a7bbf6b846e5841444acff0cd6d4/users/1ae3afa23f754392af735233b3975b4c/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:08:56.431211 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d85236b-5f71-49f6-8734-d1fd6f283f31 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:56.431757 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d85236b-5f71-49f6-8734-d1fd6f283f31 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:56.433242 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d85236b-5f71-49f6-8734-d1fd6f283f31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:56.474175 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-0d85236b-5f71-49f6-8734-d1fd6f283f31 None None] MFA Rules not processed for user `1ae3afa23f754392af735233b3975b4c`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:08:56.475705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-0d85236b-5f71-49f6-8734-d1fd6f283f31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:56.602614 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1264/2518] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:08:56 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:08:56.606140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bf9b80-7ace-4288-bc00-a28a1eada848 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:56.681345 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bf9b80-7ace-4288-bc00-a28a1eada848 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:56.687799 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706b93a7-b712-4ba5-a535-ab7cb441d240 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:56.688450 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706b93a7-b712-4ba5-a535-ab7cb441d240 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:56.688731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706b93a7-b712-4ba5-a535-ab7cb441d240 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:56.689004 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706b93a7-b712-4ba5-a535-ab7cb441d240 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:56.689674 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706b93a7-b712-4ba5-a535-ab7cb441d240 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:56.691741 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706b93a7-b712-4ba5-a535-ab7cb441d240 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:57.221953 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1255/2519] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:08:56 2022] POST /identity/v3/projects => generated 383 bytes in 616 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:08:57.224907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d85236b-5f71-49f6-8734-d1fd6f283f31 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:57.239112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d85236b-5f71-49f6-8734-d1fd6f283f31 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:57.296973 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:08:57.312100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa4f926-194a-43c3-b68c-6c0660b7c342 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:57.312708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa4f926-194a-43c3-b68c-6c0660b7c342 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:57.312983 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa4f926-194a-43c3-b68c-6c0660b7c342 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:57.313251 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa4f926-194a-43c3-b68c-6c0660b7c342 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:57.313875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfa4f926-194a-43c3-b68c-6c0660b7c342 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:57.316522 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfa4f926-194a-43c3-b68c-6c0660b7c342 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:57.384539 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1265/2520] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:08:57 2022] POST /identity/v3/users => generated 298 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:08:57.388927 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706b93a7-b712-4ba5-a535-ab7cb441d240 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:57.402764 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706b93a7-b712-4ba5-a535-ab7cb441d240 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:57.417783 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326aba65-c814-4dc9-8c2a-e9e5c6de4dc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:57.423405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326aba65-c814-4dc9-8c2a-e9e5c6de4dc3 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:57.427425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326aba65-c814-4dc9-8c2a-e9e5c6de4dc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:57.427425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326aba65-c814-4dc9-8c2a-e9e5c6de4dc3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:57.427425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326aba65-c814-4dc9-8c2a-e9e5c6de4dc3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:57.427425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326aba65-c814-4dc9-8c2a-e9e5c6de4dc3 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:57.454984 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1256/2521] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:08:57 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:08:57.458223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa4f926-194a-43c3-b68c-6c0660b7c342 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:57.472909 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa4f926-194a-43c3-b68c-6c0660b7c342 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:57.495724 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc5c8cb-05b9-4bf3-b18d-0e28e1dc7cfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:57.497182 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc5c8cb-05b9-4bf3-b18d-0e28e1dc7cfb admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:57.497915 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc5c8cb-05b9-4bf3-b18d-0e28e1dc7cfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:57.498204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc5c8cb-05b9-4bf3-b18d-0e28e1dc7cfb admin admin] PATH_INFO: `/v3/projects/b3a198a08f1d4daeb73edbdc9a614ce3/users/d01732b314a74deb8c7236f164e85a8b/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:57.524046 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc5c8cb-05b9-4bf3-b18d-0e28e1dc7cfb admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3a198a08f1d4daeb73edbdc9a614ce3, user_id=d01732b314a74deb8c7236f164e85a8b, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:57.527143 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc5c8cb-05b9-4bf3-b18d-0e28e1dc7cfb admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:57.622889 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1266/2522] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:08:57 2022] PUT /identity/v3/projects/b3a198a08f1d4daeb73edbdc9a614ce3/users/d01732b314a74deb8c7236f164e85a8b/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:08:57.626876 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326aba65-c814-4dc9-8c2a-e9e5c6de4dc3 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:57.639965 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326aba65-c814-4dc9-8c2a-e9e5c6de4dc3 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:57.731158 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa30703-0884-4fab-8fee-73c86bd85899 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:57.732177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa30703-0884-4fab-8fee-73c86bd85899 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:57.732670 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa30703-0884-4fab-8fee-73c86bd85899 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:57.733136 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa30703-0884-4fab-8fee-73c86bd85899 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:57.734662 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa30703-0884-4fab-8fee-73c86bd85899 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:57.744633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa30703-0884-4fab-8fee-73c86bd85899 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:57.765178 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1257/2523] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:08:57 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:08:57.776189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc5c8cb-05b9-4bf3-b18d-0e28e1dc7cfb admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:57.787817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc5c8cb-05b9-4bf3-b18d-0e28e1dc7cfb admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:57.793080 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c50fb6c-aa0e-42bb-96c3-7d24435bfc38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:57.793772 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c50fb6c-aa0e-42bb-96c3-7d24435bfc38 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:57.794029 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c50fb6c-aa0e-42bb-96c3-7d24435bfc38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:57.794413 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c50fb6c-aa0e-42bb-96c3-7d24435bfc38 admin admin] PATH_INFO: `/v3/projects/b3a198a08f1d4daeb73edbdc9a614ce3/users/d01732b314a74deb8c7236f164e85a8b/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:57.796536 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c50fb6c-aa0e-42bb-96c3-7d24435bfc38 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3a198a08f1d4daeb73edbdc9a614ce3, user_id=d01732b314a74deb8c7236f164e85a8b, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:57.799376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c50fb6c-aa0e-42bb-96c3-7d24435bfc38 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:57.811816 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1267/2524] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:08:57 2022] PUT /identity/v3/projects/b3a198a08f1d4daeb73edbdc9a614ce3/users/d01732b314a74deb8c7236f164e85a8b/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:08:57.823842 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4ecbb5-0620-4c30-a5ef-1a9e42d3b369 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:57.824413 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4ecbb5-0620-4c30-a5ef-1a9e42d3b369 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:57.825086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4ecbb5-0620-4c30-a5ef-1a9e42d3b369 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:57.855569 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-cb4ecbb5-0620-4c30-a5ef-1a9e42d3b369 None None] MFA Rules not processed for user `d01732b314a74deb8c7236f164e85a8b`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:08:57.859335 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-cb4ecbb5-0620-4c30-a5ef-1a9e42d3b369 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:57.872837 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:08:57.939587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1258/2525] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:08:57 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:08:57.952119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c50fb6c-aa0e-42bb-96c3-7d24435bfc38 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:57.954650 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-6c50fb6c-aa0e-42bb-96c3-7d24435bfc38 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:58.029751 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c50fb6c-aa0e-42bb-96c3-7d24435bfc38 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:58.034988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ba9111-cb45-4e00-b279-6999ace44757 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:58.035836 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ba9111-cb45-4e00-b279-6999ace44757 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:58.036182 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ba9111-cb45-4e00-b279-6999ace44757 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:58.036753 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ba9111-cb45-4e00-b279-6999ace44757 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:58.047957 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ba9111-cb45-4e00-b279-6999ace44757 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:58.049453 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ba9111-cb45-4e00-b279-6999ace44757 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:58.060171 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1268/2526] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:08:57 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:08:58.386654 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4ecbb5-0620-4c30-a5ef-1a9e42d3b369 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:58.388785 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-cb4ecbb5-0620-4c30-a5ef-1a9e42d3b369 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:58.400643 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4ecbb5-0620-4c30-a5ef-1a9e42d3b369 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:58.404702 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766fc022-51d0-4ef1-a7b1-b98b3f341927 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:58.405322 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766fc022-51d0-4ef1-a7b1-b98b3f341927 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:58.405597 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766fc022-51d0-4ef1-a7b1-b98b3f341927 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:58.405867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766fc022-51d0-4ef1-a7b1-b98b3f341927 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:58.474873 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766fc022-51d0-4ef1-a7b1-b98b3f341927 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:58.476513 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766fc022-51d0-4ef1-a7b1-b98b3f341927 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:58.487468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1259/2527] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:08:58 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:08:59.064288 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ba9111-cb45-4e00-b279-6999ace44757 service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:59.126519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ba9111-cb45-4e00-b279-6999ace44757 service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:59.131108 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f47e61-0da7-4cd0-912e-aa017fd01523 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:59.131727 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f47e61-0da7-4cd0-912e-aa017fd01523 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:59.132008 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f47e61-0da7-4cd0-912e-aa017fd01523 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:59.132279 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f47e61-0da7-4cd0-912e-aa017fd01523 admin admin] PATH_INFO: `/v3/users/1ae3afa23f754392af735233b3975b4c` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:59.133400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f47e61-0da7-4cd0-912e-aa017fd01523 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1ae3afa23f754392af735233b3975b4c)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:59.135351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f47e61-0da7-4cd0-912e-aa017fd01523 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:59.292307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:INSERT=5,keystone:DELETE=6 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:08:59.297992 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-15f47e61-0da7-4cd0-912e-aa017fd01523 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1ae3afa23f754392af735233b3975b4c'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:59.369102 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-15f47e61-0da7-4cd0-912e-aa017fd01523 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1ae3afa23f754392af735233b3975b4c'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:59.382280 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-15f47e61-0da7-4cd0-912e-aa017fd01523 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1ae3afa23f754392af735233b3975b4c'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:59.382968 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-15f47e61-0da7-4cd0-912e-aa017fd01523 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1ae3afa23f754392af735233b3975b4c'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:59.390871 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-15f47e61-0da7-4cd0-912e-aa017fd01523 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1ae3afa23f754392af735233b3975b4c'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:08:59.397044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1269/2528] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:08:59 2022] DELETE /identity/v3/users/1ae3afa23f754392af735233b3975b4c => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:08:59.401659 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e0b576-b762-4c96-95bf-bb3bd88434f6 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:59.402142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e0b576-b762-4c96-95bf-bb3bd88434f6 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:59.402260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e0b576-b762-4c96-95bf-bb3bd88434f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:59.422230 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-f4e0b576-b762-4c96-95bf-bb3bd88434f6 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:08:59.423461 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f4e0b576-b762-4c96-95bf-bb3bd88434f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:59.484842 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1260/2529] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:08:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:08:59.498955 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f47e61-0da7-4cd0-912e-aa017fd01523 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:08:59.498955 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-15f47e61-0da7-4cd0-912e-aa017fd01523 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:08:59.565393 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f47e61-0da7-4cd0-912e-aa017fd01523 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:08:59.570190 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6997160-c2c0-4c45-af62-1d96d291b253 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:08:59.570898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6997160-c2c0-4c45-af62-1d96d291b253 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:08:59.571240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6997160-c2c0-4c45-af62-1d96d291b253 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:08:59.571577 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6997160-c2c0-4c45-af62-1d96d291b253 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:08:59.591263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6997160-c2c0-4c45-af62-1d96d291b253 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:08:59.592958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6997160-c2c0-4c45-af62-1d96d291b253 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:08:59.603831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1270/2530] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:08:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:00.138997 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e0b576-b762-4c96-95bf-bb3bd88434f6 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:00.140524 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f4e0b576-b762-4c96-95bf-bb3bd88434f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:00.153609 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e0b576-b762-4c96-95bf-bb3bd88434f6 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:00.158267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42be61d1-5565-4b96-a645-5496ee99fe5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:00.158938 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42be61d1-5565-4b96-a645-5496ee99fe5b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:00.159217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42be61d1-5565-4b96-a645-5496ee99fe5b admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:00.159492 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42be61d1-5565-4b96-a645-5496ee99fe5b admin admin] PATH_INFO: `/v3/projects/ff79a7bbf6b846e5841444acff0cd6d4` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:00.160619 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42be61d1-5565-4b96-a645-5496ee99fe5b admin admin] RBAC: Authorizing `identity:delete_project(project_id=ff79a7bbf6b846e5841444acff0cd6d4)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:00.162641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42be61d1-5565-4b96-a645-5496ee99fe5b admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:00.287378 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:00.291848 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-42be61d1-5565-4b96-a645-5496ee99fe5b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ff79a7bbf6b846e5841444acff0cd6d4', 'request_id': 'req-42be61d1-5565-4b96-a645-5496ee99fe5b'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:00.368540 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-42be61d1-5565-4b96-a645-5496ee99fe5b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ff79a7bbf6b846e5841444acff0cd6d4', 'request_id': 'req-42be61d1-5565-4b96-a645-5496ee99fe5b'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:00.375987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-42be61d1-5565-4b96-a645-5496ee99fe5b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ff79a7bbf6b846e5841444acff0cd6d4', 'request_id': 'req-42be61d1-5565-4b96-a645-5496ee99fe5b'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:00.387394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-42be61d1-5565-4b96-a645-5496ee99fe5b admin admin] The token cache is being invalidate because project ff79a7bbf6b846e5841444acff0cd6d4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:09:00.388005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-42be61d1-5565-4b96-a645-5496ee99fe5b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:00.388871 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-42be61d1-5565-4b96-a645-5496ee99fe5b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:00.390452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1261/2531] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:09:00 2022] DELETE /identity/v3/projects/ff79a7bbf6b846e5841444acff0cd6d4 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:09:00.394400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6997160-c2c0-4c45-af62-1d96d291b253 service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:00.404753 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6997160-c2c0-4c45-af62-1d96d291b253 service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:00.467904 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e168546-018f-41a9-ad50-b3b9fd9dab77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:00.468526 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e168546-018f-41a9-ad50-b3b9fd9dab77 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:00.468800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e168546-018f-41a9-ad50-b3b9fd9dab77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:00.469068 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e168546-018f-41a9-ad50-b3b9fd9dab77 admin admin] PATH_INFO: `/v3/users/d01732b314a74deb8c7236f164e85a8b` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:00.470220 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e168546-018f-41a9-ad50-b3b9fd9dab77 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d01732b314a74deb8c7236f164e85a8b)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:00.472177 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e168546-018f-41a9-ad50-b3b9fd9dab77 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:00.691970 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-1e168546-018f-41a9-ad50-b3b9fd9dab77 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd01732b314a74deb8c7236f164e85a8b'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:00.757502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-1e168546-018f-41a9-ad50-b3b9fd9dab77 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd01732b314a74deb8c7236f164e85a8b'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:00.765488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:00.771122 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-1e168546-018f-41a9-ad50-b3b9fd9dab77 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd01732b314a74deb8c7236f164e85a8b'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:00.771760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-1e168546-018f-41a9-ad50-b3b9fd9dab77 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd01732b314a74deb8c7236f164e85a8b'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:00.779731 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-1e168546-018f-41a9-ad50-b3b9fd9dab77 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd01732b314a74deb8c7236f164e85a8b'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:00.785525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1271/2532] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:09:00 2022] DELETE /identity/v3/users/d01732b314a74deb8c7236f164e85a8b => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:01.205338 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42be61d1-5565-4b96-a645-5496ee99fe5b admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:01.206828 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-42be61d1-5565-4b96-a645-5496ee99fe5b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:01.223895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42be61d1-5565-4b96-a645-5496ee99fe5b admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:01.283091 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d55812-4d6f-47c3-b6c7-50cf473720fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:01.283890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d55812-4d6f-47c3-b6c7-50cf473720fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:01.284166 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d55812-4d6f-47c3-b6c7-50cf473720fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:01.284464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d55812-4d6f-47c3-b6c7-50cf473720fe admin admin] PATH_INFO: `/v3/projects/b3a198a08f1d4daeb73edbdc9a614ce3` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:01.285622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d55812-4d6f-47c3-b6c7-50cf473720fe admin admin] RBAC: Authorizing `identity:delete_project(project_id=b3a198a08f1d4daeb73edbdc9a614ce3)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:01.287544 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d55812-4d6f-47c3-b6c7-50cf473720fe admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:01.423434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-88d55812-4d6f-47c3-b6c7-50cf473720fe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b3a198a08f1d4daeb73edbdc9a614ce3', 'request_id': 'req-88d55812-4d6f-47c3-b6c7-50cf473720fe'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:01.483664 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-88d55812-4d6f-47c3-b6c7-50cf473720fe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b3a198a08f1d4daeb73edbdc9a614ce3', 'request_id': 'req-88d55812-4d6f-47c3-b6c7-50cf473720fe'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:01.490106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-88d55812-4d6f-47c3-b6c7-50cf473720fe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b3a198a08f1d4daeb73edbdc9a614ce3', 'request_id': 'req-88d55812-4d6f-47c3-b6c7-50cf473720fe'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:01.504546 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-88d55812-4d6f-47c3-b6c7-50cf473720fe admin admin] The token cache is being invalidate because project b3a198a08f1d4daeb73edbdc9a614ce3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:09:01.504839 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-88d55812-4d6f-47c3-b6c7-50cf473720fe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:01.505583 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-88d55812-4d6f-47c3-b6c7-50cf473720fe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:01.507130 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1262/2533] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:09:01 2022] DELETE /identity/v3/projects/b3a198a08f1d4daeb73edbdc9a614ce3 => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:01.512061 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3654f3-fa10-4967-9f77-92625b9d52ca None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:01.512552 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3654f3-fa10-4967-9f77-92625b9d52ca None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:01.512801 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3654f3-fa10-4967-9f77-92625b9d52ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:01.536997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-4d3654f3-fa10-4967-9f77-92625b9d52ca None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:01.538213 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-4d3654f3-fa10-4967-9f77-92625b9d52ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:01.610851 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1272/2534] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:01.618221 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7200a3a-25fe-4c6c-b704-f3e93927f331 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:01.618538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7200a3a-25fe-4c6c-b704-f3e93927f331 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:01.618817 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7200a3a-25fe-4c6c-b704-f3e93927f331 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:01.644696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-a7200a3a-25fe-4c6c-b704-f3e93927f331 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:01.645903 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a7200a3a-25fe-4c6c-b704-f3e93927f331 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:01.656194 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1263/2535] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:01.660422 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3654f3-fa10-4967-9f77-92625b9d52ca None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:01.672769 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3654f3-fa10-4967-9f77-92625b9d52ca None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:01.677019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81608981-527f-425e-b3b4-673aae7de28c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:01.677632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81608981-527f-425e-b3b4-673aae7de28c admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:01.677915 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81608981-527f-425e-b3b4-673aae7de28c admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:01.678186 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81608981-527f-425e-b3b4-673aae7de28c admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:01.678871 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81608981-527f-425e-b3b4-673aae7de28c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:09:01.679177 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81608981-527f-425e-b3b4-673aae7de28c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:01.680785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81608981-527f-425e-b3b4-673aae7de28c admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:01.694706 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1273/2536] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:09:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:09:01.698318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7200a3a-25fe-4c6c-b704-f3e93927f331 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:01.708952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7200a3a-25fe-4c6c-b704-f3e93927f331 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:01.713622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcf01c2-4980-4c99-aac0-3f05920f7af4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:01.714216 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcf01c2-4980-4c99-aac0-3f05920f7af4 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:01.714521 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcf01c2-4980-4c99-aac0-3f05920f7af4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:01.714804 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcf01c2-4980-4c99-aac0-3f05920f7af4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:01.716207 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcf01c2-4980-4c99-aac0-3f05920f7af4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:01.718305 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcf01c2-4980-4c99-aac0-3f05920f7af4 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:01.794077 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1264/2537] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:09:01 2022] POST /identity/v3/projects => generated 371 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:09:01.798666 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81608981-527f-425e-b3b4-673aae7de28c admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:01.809153 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81608981-527f-425e-b3b4-673aae7de28c admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:01.868583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3edcc1-d999-48c4-a39d-7564fc3ae66a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:01.869212 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3edcc1-d999-48c4-a39d-7564fc3ae66a admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:01.869488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3edcc1-d999-48c4-a39d-7564fc3ae66a admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:01.869761 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3edcc1-d999-48c4-a39d-7564fc3ae66a admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:01.870490 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3edcc1-d999-48c4-a39d-7564fc3ae66a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:01.872645 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3edcc1-d999-48c4-a39d-7564fc3ae66a admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:01.986855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1274/2538] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:09:01 2022] POST /identity/v3/users => generated 286 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:09:01.991179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcf01c2-4980-4c99-aac0-3f05920f7af4 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:02.002422 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcf01c2-4980-4c99-aac0-3f05920f7af4 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:02.006990 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7891a14f-d6ad-4dc6-95c8-58cae91d39a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:02.007615 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7891a14f-d6ad-4dc6-95c8-58cae91d39a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:02.007892 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7891a14f-d6ad-4dc6-95c8-58cae91d39a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:02.008165 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7891a14f-d6ad-4dc6-95c8-58cae91d39a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:02.008855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7891a14f-d6ad-4dc6-95c8-58cae91d39a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:02.010465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7891a14f-d6ad-4dc6-95c8-58cae91d39a5 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:02.026753 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1265/2539] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:01 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:09:02.029681 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3edcc1-d999-48c4-a39d-7564fc3ae66a admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:02.039763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3edcc1-d999-48c4-a39d-7564fc3ae66a admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:02.044516 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f797c86-0a9c-41ca-9ac0-442525e5ebaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:02.045103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f797c86-0a9c-41ca-9ac0-442525e5ebaf admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:02.045675 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f797c86-0a9c-41ca-9ac0-442525e5ebaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:02.045675 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f797c86-0a9c-41ca-9ac0-442525e5ebaf admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:02.046309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f797c86-0a9c-41ca-9ac0-442525e5ebaf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:02.048010 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f797c86-0a9c-41ca-9ac0-442525e5ebaf admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:02.059706 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1275/2540] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:02 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:09:02.064156 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7891a14f-d6ad-4dc6-95c8-58cae91d39a5 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:02.074187 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7891a14f-d6ad-4dc6-95c8-58cae91d39a5 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:02.079641 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3a946c-7fce-479b-909d-63e3a761dbce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:02.080285 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3a946c-7fce-479b-909d-63e3a761dbce admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:02.080583 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3a946c-7fce-479b-909d-63e3a761dbce admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:02.080852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3a946c-7fce-479b-909d-63e3a761dbce admin admin] PATH_INFO: `/v3/projects/3ef0c5d7373640d6bb149b011f074ae3/users/2af73db67a574429b59df945eccf74af/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:02.095724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3a946c-7fce-479b-909d-63e3a761dbce admin admin] RBAC: Authorizing `identity:create_grant(project_id=3ef0c5d7373640d6bb149b011f074ae3, user_id=2af73db67a574429b59df945eccf74af, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:02.098448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3a946c-7fce-479b-909d-63e3a761dbce admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:02.164915 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:DELETE=11,keystone:SELECT=85,keystone:INSERT=4 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:02.169677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1266/2541] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:09:02 2022] PUT /identity/v3/projects/3ef0c5d7373640d6bb149b011f074ae3/users/2af73db67a574429b59df945eccf74af/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:02.174785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a3bb8d-46f3-468c-8704-73559825f28a None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:02.182640 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a3bb8d-46f3-468c-8704-73559825f28a None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:02.182640 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29a3bb8d-46f3-468c-8704-73559825f28a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:02.195450 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-29a3bb8d-46f3-468c-8704-73559825f28a None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:02.196641 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-29a3bb8d-46f3-468c-8704-73559825f28a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:02.238907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:02.257181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1276/2542] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:02.267908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3a946c-7fce-479b-909d-63e3a761dbce admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:02.271442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-4b3a946c-7fce-479b-909d-63e3a761dbce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:02.331537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3a946c-7fce-479b-909d-63e3a761dbce admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:02.335095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601ab4a3-93e7-4416-a8ce-7e6944493af3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:02.335696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601ab4a3-93e7-4416-a8ce-7e6944493af3 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:02.335976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601ab4a3-93e7-4416-a8ce-7e6944493af3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:02.336248 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601ab4a3-93e7-4416-a8ce-7e6944493af3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:02.345431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-601ab4a3-93e7-4416-a8ce-7e6944493af3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:02.346887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-601ab4a3-93e7-4416-a8ce-7e6944493af3 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:02.356575 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1267/2543] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:04.047934 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f416d16b-7e28-47e2-aa2d-c23cf59d1607 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:04.048471 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f416d16b-7e28-47e2-aa2d-c23cf59d1607 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:04.048566 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f416d16b-7e28-47e2-aa2d-c23cf59d1607 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:04.094504 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-f416d16b-7e28-47e2-aa2d-c23cf59d1607 None None] MFA Rules not processed for user `2af73db67a574429b59df945eccf74af`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:04.095660 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f416d16b-7e28-47e2-aa2d-c23cf59d1607 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:04.204277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1277/2544] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:04 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:04.209488 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601ab4a3-93e7-4416-a8ce-7e6944493af3 service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:04.221722 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601ab4a3-93e7-4416-a8ce-7e6944493af3 service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:04.228821 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43308070-4119-4d3d-b7dd-5299b4a170e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:04.234583 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43308070-4119-4d3d-b7dd-5299b4a170e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:04.234583 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43308070-4119-4d3d-b7dd-5299b4a170e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:04.234583 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43308070-4119-4d3d-b7dd-5299b4a170e0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:04.234583 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43308070-4119-4d3d-b7dd-5299b4a170e0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:04.234583 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43308070-4119-4d3d-b7dd-5299b4a170e0 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:04.355496 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1268/2545] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:09:04 2022] POST /identity/v3/projects => generated 373 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:09:04.360790 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f416d16b-7e28-47e2-aa2d-c23cf59d1607 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:04.381096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f416d16b-7e28-47e2-aa2d-c23cf59d1607 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:04.472490 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83962883-a531-4274-aaf8-18ba02a74845 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:04.473104 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83962883-a531-4274-aaf8-18ba02a74845 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:04.478768 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83962883-a531-4274-aaf8-18ba02a74845 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:04.479059 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83962883-a531-4274-aaf8-18ba02a74845 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:04.479736 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83962883-a531-4274-aaf8-18ba02a74845 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:04.483433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83962883-a531-4274-aaf8-18ba02a74845 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:04.582506 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1278/2546] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:09:04 2022] POST /identity/v3/users => generated 293 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:09:04.587682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43308070-4119-4d3d-b7dd-5299b4a170e0 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:04.599790 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43308070-4119-4d3d-b7dd-5299b4a170e0 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:04.605115 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8399d05d-baa1-4293-8a95-2b6c904aa90a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:04.606881 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8399d05d-baa1-4293-8a95-2b6c904aa90a admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:04.607592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8399d05d-baa1-4293-8a95-2b6c904aa90a admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:04.608526 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8399d05d-baa1-4293-8a95-2b6c904aa90a admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:04.609701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8399d05d-baa1-4293-8a95-2b6c904aa90a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:04.611632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8399d05d-baa1-4293-8a95-2b6c904aa90a admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:04.624947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1269/2547] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:04 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:09:04.630112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83962883-a531-4274-aaf8-18ba02a74845 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:04.644699 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83962883-a531-4274-aaf8-18ba02a74845 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:04.652649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7e4974-2617-4b71-80b0-ddd9d74c5f3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:04.653535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7e4974-2617-4b71-80b0-ddd9d74c5f3d admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:04.653959 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7e4974-2617-4b71-80b0-ddd9d74c5f3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:04.654324 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7e4974-2617-4b71-80b0-ddd9d74c5f3d admin admin] PATH_INFO: `/v3/projects/e001a2e371f7483ab56127aca94971fc/users/5c606509ef13421eafd1e54d556b9c8e/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:04.671096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7e4974-2617-4b71-80b0-ddd9d74c5f3d admin admin] RBAC: Authorizing `identity:create_grant(project_id=e001a2e371f7483ab56127aca94971fc, user_id=5c606509ef13421eafd1e54d556b9c8e, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:04.673787 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7e4974-2617-4b71-80b0-ddd9d74c5f3d admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:04.804824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1279/2548] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:09:04 2022] PUT /identity/v3/projects/e001a2e371f7483ab56127aca94971fc/users/5c606509ef13421eafd1e54d556b9c8e/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:09:04.809932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8399d05d-baa1-4293-8a95-2b6c904aa90a admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:04.820418 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8399d05d-baa1-4293-8a95-2b6c904aa90a admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:04.883323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904e5260-a621-4297-a546-7fe71a7452e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:04.884028 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904e5260-a621-4297-a546-7fe71a7452e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:04.884306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904e5260-a621-4297-a546-7fe71a7452e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:04.884576 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904e5260-a621-4297-a546-7fe71a7452e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:04.885248 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904e5260-a621-4297-a546-7fe71a7452e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:04.886913 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904e5260-a621-4297-a546-7fe71a7452e5 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:04.901956 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1270/2549] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:04 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:09:04.903469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7e4974-2617-4b71-80b0-ddd9d74c5f3d admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:04.922230 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7e4974-2617-4b71-80b0-ddd9d74c5f3d admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:04.928740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73eb2c95-b0d7-4c7c-b34d-c4b3041912f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:04.929419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73eb2c95-b0d7-4c7c-b34d-c4b3041912f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:04.929696 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73eb2c95-b0d7-4c7c-b34d-c4b3041912f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:04.929968 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73eb2c95-b0d7-4c7c-b34d-c4b3041912f5 admin admin] PATH_INFO: `/v3/projects/e001a2e371f7483ab56127aca94971fc/users/5c606509ef13421eafd1e54d556b9c8e/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:04.932555 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73eb2c95-b0d7-4c7c-b34d-c4b3041912f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e001a2e371f7483ab56127aca94971fc, user_id=5c606509ef13421eafd1e54d556b9c8e, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:04.936377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73eb2c95-b0d7-4c7c-b34d-c4b3041912f5 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:04.954892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1280/2550] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:09:04 2022] PUT /identity/v3/projects/e001a2e371f7483ab56127aca94971fc/users/5c606509ef13421eafd1e54d556b9c8e/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:09:06.406683 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5302d8f8-5a37-4461-b393-58769e683f8e None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:06.407152 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5302d8f8-5a37-4461-b393-58769e683f8e None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:06.407430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5302d8f8-5a37-4461-b393-58769e683f8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:06.451122 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-5302d8f8-5a37-4461-b393-58769e683f8e None None] MFA Rules not processed for user `5c606509ef13421eafd1e54d556b9c8e`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:06.452351 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-5302d8f8-5a37-4461-b393-58769e683f8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:06.522955 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:06.541744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1271/2551] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:06 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:06.553588 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73eb2c95-b0d7-4c7c-b34d-c4b3041912f5 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:06.555317 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-73eb2c95-b0d7-4c7c-b34d-c4b3041912f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:06.588436 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:06.614186 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73eb2c95-b0d7-4c7c-b34d-c4b3041912f5 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:06.618125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc6193a-f4ce-4cb9-ba3a-dc1bcdc9af36 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:06.618767 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc6193a-f4ce-4cb9-ba3a-dc1bcdc9af36 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:06.619048 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc6193a-f4ce-4cb9-ba3a-dc1bcdc9af36 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:06.619321 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc6193a-f4ce-4cb9-ba3a-dc1bcdc9af36 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:06.688219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc6193a-f4ce-4cb9-ba3a-dc1bcdc9af36 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:06.689660 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc6193a-f4ce-4cb9-ba3a-dc1bcdc9af36 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:06.699073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1281/2552] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:06 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:08.274772 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d76c5b-465b-41e0-917a-c918a31444b8 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:08.275401 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d76c5b-465b-41e0-917a-c918a31444b8 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:08.275401 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d76c5b-465b-41e0-917a-c918a31444b8 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:08.276887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1272/2553] 10.222.0.62 () {54 vars in 921 bytes} [Fri Sep 23 14:09:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:09:08.283245 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b0f403-8f56-4a03-96c1-f1758e36077a None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:08.283731 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b0f403-8f56-4a03-96c1-f1758e36077a None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:08.283827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b0f403-8f56-4a03-96c1-f1758e36077a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:08.305165 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-63b0f403-8f56-4a03-96c1-f1758e36077a None None] MFA Rules not processed for user `f01ed94dbb384a178ad0a45a6f3d8c17`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:08.306884 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-63b0f403-8f56-4a03-96c1-f1758e36077a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:08.497466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d76c5b-465b-41e0-917a-c918a31444b8 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:08.501803 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1282/2554] 10.222.0.62 () {58 vars in 1018 bytes} [Fri Sep 23 14:09:08 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:08.510551 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d76c5b-465b-41e0-917a-c918a31444b8 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:08.514726 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b0f403-8f56-4a03-96c1-f1758e36077a None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:08.515109 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a78dee6-f42a-4104-8139-9aaaaf10b208 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:08.515885 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a78dee6-f42a-4104-8139-9aaaaf10b208 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:08.516159 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a78dee6-f42a-4104-8139-9aaaaf10b208 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:08.516425 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a78dee6-f42a-4104-8139-9aaaaf10b208 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:08.518498 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-63b0f403-8f56-4a03-96c1-f1758e36077a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:08.528758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a78dee6-f42a-4104-8139-9aaaaf10b208 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:08.533922 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a78dee6-f42a-4104-8139-9aaaaf10b208 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:08.540621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1273/2555] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:08 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:09:08.594917 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b0f403-8f56-4a03-96c1-f1758e36077a None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:08.598845 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfca55a-4ab5-4c65-98b6-c8032554b443 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'acdf4a7bd608492390176dcc42ca1267', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:08.599450 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfca55a-4ab5-4c65-98b6-c8032554b443 service glance] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:08.599723 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfca55a-4ab5-4c65-98b6-c8032554b443 service glance] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:08.599996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfca55a-4ab5-4c65-98b6-c8032554b443 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:08.613173 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfca55a-4ab5-4c65-98b6-c8032554b443 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:08.614759 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfca55a-4ab5-4c65-98b6-c8032554b443 service glance] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:08.626032 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1283/2556] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:08 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:09:08.834446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addf0021-56fb-421b-b7d5-4653a263589f None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:08.834732 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addf0021-56fb-421b-b7d5-4653a263589f None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:08.835010 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addf0021-56fb-421b-b7d5-4653a263589f None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:08.836477 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1274/2557] 10.222.0.62 () {54 vars in 921 bytes} [Fri Sep 23 14:09:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:09:08.841521 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0bbaa4-f468-4a68-91cf-8556a1186f6b None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:08.841807 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0bbaa4-f468-4a68-91cf-8556a1186f6b None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:08.842072 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0bbaa4-f468-4a68-91cf-8556a1186f6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:08.862391 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-aa0bbaa4-f468-4a68-91cf-8556a1186f6b None None] MFA Rules not processed for user `580a35627c234a25aa04f5d376177ca5`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:08.863605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-aa0bbaa4-f468-4a68-91cf-8556a1186f6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:08.877519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:08.928925 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1284/2558] 10.222.0.62 () {58 vars in 1018 bytes} [Fri Sep 23 14:09:08 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:08.942049 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addf0021-56fb-421b-b7d5-4653a263589f None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:08.955235 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addf0021-56fb-421b-b7d5-4653a263589f None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:08.963922 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4058e477-f54b-481b-a5ba-c9fa786f19c5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:08.963922 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4058e477-f54b-481b-a5ba-c9fa786f19c5 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:08.964472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4058e477-f54b-481b-a5ba-c9fa786f19c5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:08.964917 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4058e477-f54b-481b-a5ba-c9fa786f19c5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:08.973393 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4058e477-f54b-481b-a5ba-c9fa786f19c5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:08.975172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4058e477-f54b-481b-a5ba-c9fa786f19c5 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:08.984109 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1275/2559] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:08 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:09.120592 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248023e7-c5ca-49c7-95b1-ed0d77d3a06d None None] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:09.120888 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248023e7-c5ca-49c7-95b1-ed0d77d3a06d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:09.121163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248023e7-c5ca-49c7-95b1-ed0d77d3a06d None None] PATH_INFO: `/` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:09.123768 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1285/2560] 10.222.0.62 () {54 vars in 921 bytes} [Fri Sep 23 14:09:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:09:09.130365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326377ff-c6b1-47f0-941a-9ca44f2e3e0f None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:09.130978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326377ff-c6b1-47f0-941a-9ca44f2e3e0f None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:09.131313 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326377ff-c6b1-47f0-941a-9ca44f2e3e0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:09.158965 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-326377ff-c6b1-47f0-941a-9ca44f2e3e0f None None] MFA Rules not processed for user `f01ed94dbb384a178ad0a45a6f3d8c17`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:09.160269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-326377ff-c6b1-47f0-941a-9ca44f2e3e0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:09.172223 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1276/2561] 10.222.0.62 () {58 vars in 1018 bytes} [Fri Sep 23 14:09:09 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:09.185654 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248023e7-c5ca-49c7-95b1-ed0d77d3a06d None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:09.187972 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-248023e7-c5ca-49c7-95b1-ed0d77d3a06d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:09.212898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248023e7-c5ca-49c7-95b1-ed0d77d3a06d None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:09.216690 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8132af1-d566-4dd3-a9eb-1a371d65582c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:09.219217 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8132af1-d566-4dd3-a9eb-1a371d65582c service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:09.219217 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8132af1-d566-4dd3-a9eb-1a371d65582c service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:09.219217 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8132af1-d566-4dd3-a9eb-1a371d65582c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:09.235585 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8132af1-d566-4dd3-a9eb-1a371d65582c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:09.237356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8132af1-d566-4dd3-a9eb-1a371d65582c service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:09.249442 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1286/2562] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:12.073430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326377ff-c6b1-47f0-941a-9ca44f2e3e0f None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:12.086683 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326377ff-c6b1-47f0-941a-9ca44f2e3e0f None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:12.096849 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe68f59-3694-4526-a680-013f676afb8f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:12.097547 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe68f59-3694-4526-a680-013f676afb8f service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:12.097894 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe68f59-3694-4526-a680-013f676afb8f service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:12.098224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe68f59-3694-4526-a680-013f676afb8f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:12.100764 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ebe68f59-3694-4526-a680-013f676afb8f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:12.121821 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8132af1-d566-4dd3-a9eb-1a371d65582c service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:12.133093 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8132af1-d566-4dd3-a9eb-1a371d65582c service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:12.137294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c301ce-adc5-4729-9282-04569f883bc4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:12.137891 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c301ce-adc5-4729-9282-04569f883bc4 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:12.138227 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c301ce-adc5-4729-9282-04569f883bc4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:12.138529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c301ce-adc5-4729-9282-04569f883bc4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:12.140800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-47c301ce-adc5-4729-9282-04569f883bc4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:12.181362 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe68f59-3694-4526-a680-013f676afb8f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:12.183082 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe68f59-3694-4526-a680-013f676afb8f service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:12.194426 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1277/2563] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:12 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:12.218306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c301ce-adc5-4729-9282-04569f883bc4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:12.220553 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47c301ce-adc5-4729-9282-04569f883bc4 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:12.233066 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1287/2564] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:09:12.720722 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71ce0b2-bfc7-4bca-85eb-ee4b43501536 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:12.720913 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71ce0b2-bfc7-4bca-85eb-ee4b43501536 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:12.721156 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71ce0b2-bfc7-4bca-85eb-ee4b43501536 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:12.762674 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-e71ce0b2-bfc7-4bca-85eb-ee4b43501536 None None] MFA Rules not processed for user `b471d768cd814ff8b4f6bd35d21e24a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:12.780888 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e71ce0b2-bfc7-4bca-85eb-ee4b43501536 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:12.829902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1278/2565] 10.222.0.62 () {58 vars in 1009 bytes} [Fri Sep 23 14:09:12 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:12.843014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c301ce-adc5-4729-9282-04569f883bc4 service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:12.844549 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-47c301ce-adc5-4729-9282-04569f883bc4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:12.914902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c301ce-adc5-4729-9282-04569f883bc4 service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:12.920875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cdd3bd-012d-4465-a556-6a1da315c63b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b9269509f1d454eab7a3008c2ba0039', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:12.921483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cdd3bd-012d-4465-a556-6a1da315c63b service swift] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:12.921755 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cdd3bd-012d-4465-a556-6a1da315c63b service swift] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:12.922314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cdd3bd-012d-4465-a556-6a1da315c63b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:12.931222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cdd3bd-012d-4465-a556-6a1da315c63b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:12.933069 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77cdd3bd-012d-4465-a556-6a1da315c63b service swift] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:12.942382 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1288/2566] 10.222.0.62 () {60 vars in 1384 bytes} [Fri Sep 23 14:09:12 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 23 14:09:17.437468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7386a35-a650-43af-b9b3-19a72f6b5101 None None] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:17.441453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7386a35-a650-43af-b9b3-19a72f6b5101 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:17.441453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7386a35-a650-43af-b9b3-19a72f6b5101 None None] PATH_INFO: `/` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:17.441453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1279/2567] 10.222.0.62 () {54 vars in 921 bytes} [Fri Sep 23 14:09:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 23 14:09:17.469924 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb612248-634f-4bad-87cd-c2e7b786ff94 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:17.475982 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb612248-634f-4bad-87cd-c2e7b786ff94 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:17.475982 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb612248-634f-4bad-87cd-c2e7b786ff94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:17.495240 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-bb612248-634f-4bad-87cd-c2e7b786ff94 None None] MFA Rules not processed for user `ce249d4a99c34891b6ba941f57a14dd4`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:17.496682 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-bb612248-634f-4bad-87cd-c2e7b786ff94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:17.537642 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:17.578044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1289/2568] 10.222.0.62 () {58 vars in 1018 bytes} [Fri Sep 23 14:09:17 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:17.585873 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7386a35-a650-43af-b9b3-19a72f6b5101 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:17.587892 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a7386a35-a650-43af-b9b3-19a72f6b5101 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:17.612207 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7386a35-a650-43af-b9b3-19a72f6b5101 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:17.617699 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e393433a-8fb2-4760-aeb1-ebb7a2bd97b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:17.618586 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e393433a-8fb2-4760-aeb1-ebb7a2bd97b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:17.618964 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e393433a-8fb2-4760-aeb1-ebb7a2bd97b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:17.626402 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e393433a-8fb2-4760-aeb1-ebb7a2bd97b7 admin admin] PATH_INFO: `/v3/users/0db30f990aae49fabf9b8b19cd019de9` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:17.626402 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e393433a-8fb2-4760-aeb1-ebb7a2bd97b7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0db30f990aae49fabf9b8b19cd019de9)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:17.626402 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e393433a-8fb2-4760-aeb1-ebb7a2bd97b7 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:17.640658 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb612248-634f-4bad-87cd-c2e7b786ff94 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:17.642354 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-bb612248-634f-4bad-87cd-c2e7b786ff94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:17.673954 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb612248-634f-4bad-87cd-c2e7b786ff94 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:17.675746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172851a5-7e13-4f20-b845-5a8ba92374db service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce249d4a99c34891b6ba941f57a14dd4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:17.679707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172851a5-7e13-4f20-b845-5a8ba92374db service placement] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:17.679707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172851a5-7e13-4f20-b845-5a8ba92374db service placement] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:17.679707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172851a5-7e13-4f20-b845-5a8ba92374db service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:17.691613 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172851a5-7e13-4f20-b845-5a8ba92374db service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:17.696306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172851a5-7e13-4f20-b845-5a8ba92374db service placement] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:17.707553 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1290/2569] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:17 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:17.957661 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e393433a-8fb2-4760-aeb1-ebb7a2bd97b7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0db30f990aae49fabf9b8b19cd019de9'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:18.048518 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e393433a-8fb2-4760-aeb1-ebb7a2bd97b7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0db30f990aae49fabf9b8b19cd019de9'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:18.050520 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e393433a-8fb2-4760-aeb1-ebb7a2bd97b7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0db30f990aae49fabf9b8b19cd019de9'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:18.053501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:18.069943 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e393433a-8fb2-4760-aeb1-ebb7a2bd97b7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0db30f990aae49fabf9b8b19cd019de9'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:18.083625 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e393433a-8fb2-4760-aeb1-ebb7a2bd97b7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0db30f990aae49fabf9b8b19cd019de9'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:18.092199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1280/2570] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:09:17 2022] DELETE /identity/v3/users/0db30f990aae49fabf9b8b19cd019de9 => generated 0 bytes in 506 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:18.587883 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172851a5-7e13-4f20-b845-5a8ba92374db service placement] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:18.601676 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172851a5-7e13-4f20-b845-5a8ba92374db service placement] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:18.754234 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f7d8d0-b9c6-4f72-8853-ae0cb1523db1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:18.755087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f7d8d0-b9c6-4f72-8853-ae0cb1523db1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:18.755409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f7d8d0-b9c6-4f72-8853-ae0cb1523db1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:18.755644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f7d8d0-b9c6-4f72-8853-ae0cb1523db1 admin admin] PATH_INFO: `/v3/projects/482490c45ac64607bba9ad021f2effbf` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:18.757037 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f7d8d0-b9c6-4f72-8853-ae0cb1523db1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=482490c45ac64607bba9ad021f2effbf)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:18.759501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f7d8d0-b9c6-4f72-8853-ae0cb1523db1 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:18.926541 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-94f7d8d0-b9c6-4f72-8853-ae0cb1523db1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '482490c45ac64607bba9ad021f2effbf', 'request_id': 'req-94f7d8d0-b9c6-4f72-8853-ae0cb1523db1'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:19.007298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-94f7d8d0-b9c6-4f72-8853-ae0cb1523db1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '482490c45ac64607bba9ad021f2effbf', 'request_id': 'req-94f7d8d0-b9c6-4f72-8853-ae0cb1523db1'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:19.017836 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-94f7d8d0-b9c6-4f72-8853-ae0cb1523db1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '482490c45ac64607bba9ad021f2effbf', 'request_id': 'req-94f7d8d0-b9c6-4f72-8853-ae0cb1523db1'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:19.034817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-94f7d8d0-b9c6-4f72-8853-ae0cb1523db1 admin admin] The token cache is being invalidate because project 482490c45ac64607bba9ad021f2effbf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:09:19.035123 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-94f7d8d0-b9c6-4f72-8853-ae0cb1523db1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:19.035944 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-94f7d8d0-b9c6-4f72-8853-ae0cb1523db1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:19.037930 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1291/2571] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:09:18 2022] DELETE /identity/v3/projects/482490c45ac64607bba9ad021f2effbf => generated 0 bytes in 453 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:19.043069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365b8af-739f-488c-8300-0099b45e56a5 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:19.043598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365b8af-739f-488c-8300-0099b45e56a5 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:19.043598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365b8af-739f-488c-8300-0099b45e56a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:19.069399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-f365b8af-739f-488c-8300-0099b45e56a5 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:19.070779 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f365b8af-739f-488c-8300-0099b45e56a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:19.165527 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1281/2572] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:19.174039 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7900c09e-2c48-4943-8377-6d4160c865bf None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:19.174323 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7900c09e-2c48-4943-8377-6d4160c865bf None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:19.174815 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7900c09e-2c48-4943-8377-6d4160c865bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:19.231371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-7900c09e-2c48-4943-8377-6d4160c865bf None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:19.232792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-7900c09e-2c48-4943-8377-6d4160c865bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:19.248177 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1292/2573] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:19.251080 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f365b8af-739f-488c-8300-0099b45e56a5 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:19.263095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f365b8af-739f-488c-8300-0099b45e56a5 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:19.289387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a38724e-8978-492c-91a3-b52e2b0aab72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:19.289915 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a38724e-8978-492c-91a3-b52e2b0aab72 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:19.290198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a38724e-8978-492c-91a3-b52e2b0aab72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:19.290518 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a38724e-8978-492c-91a3-b52e2b0aab72 admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:19.291220 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a38724e-8978-492c-91a3-b52e2b0aab72 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:09:19.291528 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a38724e-8978-492c-91a3-b52e2b0aab72 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:19.293150 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a38724e-8978-492c-91a3-b52e2b0aab72 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:19.315526 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1282/2574] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:09:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:09:19.317174 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7900c09e-2c48-4943-8377-6d4160c865bf None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:19.331040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7900c09e-2c48-4943-8377-6d4160c865bf None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:19.337965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c192b1bb-d6be-4459-9da3-504bd86cc695 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:19.338773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c192b1bb-d6be-4459-9da3-504bd86cc695 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:19.339258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c192b1bb-d6be-4459-9da3-504bd86cc695 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:19.339627 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c192b1bb-d6be-4459-9da3-504bd86cc695 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:19.341608 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c192b1bb-d6be-4459-9da3-504bd86cc695 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:19.344175 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c192b1bb-d6be-4459-9da3-504bd86cc695 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:19.439746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1293/2575] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:09:19 2022] POST /identity/v3/projects => generated 381 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:09:19.445026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a38724e-8978-492c-91a3-b52e2b0aab72 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:19.454407 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a38724e-8978-492c-91a3-b52e2b0aab72 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:19.525217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdd8844-e8cb-4eff-b5ee-e416fdcdb310 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:19.525912 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdd8844-e8cb-4eff-b5ee-e416fdcdb310 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:19.526196 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdd8844-e8cb-4eff-b5ee-e416fdcdb310 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:19.526565 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdd8844-e8cb-4eff-b5ee-e416fdcdb310 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:19.531632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bdd8844-e8cb-4eff-b5ee-e416fdcdb310 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:19.531632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bdd8844-e8cb-4eff-b5ee-e416fdcdb310 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:19.622027 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1283/2576] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:09:19 2022] POST /identity/v3/users => generated 291 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:09:19.626904 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c192b1bb-d6be-4459-9da3-504bd86cc695 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:19.644135 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c192b1bb-d6be-4459-9da3-504bd86cc695 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:19.650436 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8614b0-0606-4f47-ae84-6573628fe008 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:19.651766 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8614b0-0606-4f47-ae84-6573628fe008 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:19.652454 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8614b0-0606-4f47-ae84-6573628fe008 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:19.653009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8614b0-0606-4f47-ae84-6573628fe008 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:19.654378 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8614b0-0606-4f47-ae84-6573628fe008 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:19.656784 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8614b0-0606-4f47-ae84-6573628fe008 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:19.669057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1294/2577] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:19 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:09:19.675085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdd8844-e8cb-4eff-b5ee-e416fdcdb310 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:19.687292 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdd8844-e8cb-4eff-b5ee-e416fdcdb310 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:19.693762 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14acd233-4634-4e8d-a64c-60c5a45f534d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:19.694564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14acd233-4634-4e8d-a64c-60c5a45f534d admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:19.694942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14acd233-4634-4e8d-a64c-60c5a45f534d admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:19.695473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14acd233-4634-4e8d-a64c-60c5a45f534d admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:19.696293 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14acd233-4634-4e8d-a64c-60c5a45f534d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:19.698011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14acd233-4634-4e8d-a64c-60c5a45f534d admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:19.711235 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1284/2578] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:19 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:09:19.715680 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8614b0-0606-4f47-ae84-6573628fe008 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:19.734182 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8614b0-0606-4f47-ae84-6573628fe008 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:19.739260 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0805608e-04d6-496c-8bd9-5390329c1ae2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:19.739927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0805608e-04d6-496c-8bd9-5390329c1ae2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:19.740246 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0805608e-04d6-496c-8bd9-5390329c1ae2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:19.740517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0805608e-04d6-496c-8bd9-5390329c1ae2 admin admin] PATH_INFO: `/v3/projects/3b85a316ecef43f7bb9769abc1e24adf/users/72eb449f379b45c89074100e07c2ac1a/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:19.752563 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:19.759669 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0805608e-04d6-496c-8bd9-5390329c1ae2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3b85a316ecef43f7bb9769abc1e24adf, user_id=72eb449f379b45c89074100e07c2ac1a, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:19.762473 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0805608e-04d6-496c-8bd9-5390329c1ae2 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:19.898153 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1295/2579] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:09:19 2022] PUT /identity/v3/projects/3b85a316ecef43f7bb9769abc1e24adf/users/72eb449f379b45c89074100e07c2ac1a/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:19.910527 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2726ab9b-f1a0-4810-a0df-c4d242abb3ec None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:19.910834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2726ab9b-f1a0-4810-a0df-c4d242abb3ec None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:19.911242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2726ab9b-f1a0-4810-a0df-c4d242abb3ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:19.944091 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-2726ab9b-f1a0-4810-a0df-c4d242abb3ec None None] MFA Rules not processed for user `72eb449f379b45c89074100e07c2ac1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:19.945352 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2726ab9b-f1a0-4810-a0df-c4d242abb3ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:19.977800 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:20.052767 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1285/2580] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:19 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:09:20.058676 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0805608e-04d6-496c-8bd9-5390329c1ae2 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:20.159376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0805608e-04d6-496c-8bd9-5390329c1ae2 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:20.164779 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9995333d-37b6-4a06-a9cf-1df77399a2ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:20.165398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9995333d-37b6-4a06-a9cf-1df77399a2ed admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:20.165669 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9995333d-37b6-4a06-a9cf-1df77399a2ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:20.165944 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9995333d-37b6-4a06-a9cf-1df77399a2ed admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:20.166612 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9995333d-37b6-4a06-a9cf-1df77399a2ed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:20.168683 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9995333d-37b6-4a06-a9cf-1df77399a2ed admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:20.383923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1296/2581] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:09:20 2022] POST /identity/v3/projects => generated 379 bytes in 326 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:09:20.387596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2726ab9b-f1a0-4810-a0df-c4d242abb3ec None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:20.399842 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2726ab9b-f1a0-4810-a0df-c4d242abb3ec None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:20.460907 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23088600-eb57-4a4b-af60-e821cc11d91d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:20.461523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23088600-eb57-4a4b-af60-e821cc11d91d admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:20.461799 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23088600-eb57-4a4b-af60-e821cc11d91d admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:20.462069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23088600-eb57-4a4b-af60-e821cc11d91d admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:20.462903 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23088600-eb57-4a4b-af60-e821cc11d91d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:20.464916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23088600-eb57-4a4b-af60-e821cc11d91d admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:20.587390 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1286/2582] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:09:20 2022] POST /identity/v3/users => generated 296 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:09:20.591901 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9995333d-37b6-4a06-a9cf-1df77399a2ed admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:20.605065 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9995333d-37b6-4a06-a9cf-1df77399a2ed admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:20.612626 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353efe19-8656-40d5-9252-4b6295ba20b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:20.613448 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353efe19-8656-40d5-9252-4b6295ba20b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:20.613784 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353efe19-8656-40d5-9252-4b6295ba20b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:20.614062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353efe19-8656-40d5-9252-4b6295ba20b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:20.614749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-353efe19-8656-40d5-9252-4b6295ba20b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:20.616490 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-353efe19-8656-40d5-9252-4b6295ba20b7 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:20.647846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1297/2583] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:20 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:09:20.651605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23088600-eb57-4a4b-af60-e821cc11d91d admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:20.666721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23088600-eb57-4a4b-af60-e821cc11d91d admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:20.671266 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0251a0be-ad0a-405c-87f8-115a51f17f44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:20.673060 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0251a0be-ad0a-405c-87f8-115a51f17f44 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:20.673350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0251a0be-ad0a-405c-87f8-115a51f17f44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:20.673628 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0251a0be-ad0a-405c-87f8-115a51f17f44 admin admin] PATH_INFO: `/v3/projects/288a1a257fa148a1b79a3bf3e7a77478/users/91bca22400ca4a95a9de66eb60051175/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:20.708579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0251a0be-ad0a-405c-87f8-115a51f17f44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=288a1a257fa148a1b79a3bf3e7a77478, user_id=91bca22400ca4a95a9de66eb60051175, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:20.711945 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0251a0be-ad0a-405c-87f8-115a51f17f44 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:20.799697 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1287/2584] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:09:20 2022] PUT /identity/v3/projects/288a1a257fa148a1b79a3bf3e7a77478/users/91bca22400ca4a95a9de66eb60051175/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:20.803704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353efe19-8656-40d5-9252-4b6295ba20b7 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:20.819679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353efe19-8656-40d5-9252-4b6295ba20b7 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:20.890415 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d472eafe-b836-4b0c-bbe1-498c60efc3b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:20.891059 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d472eafe-b836-4b0c-bbe1-498c60efc3b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:20.891338 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d472eafe-b836-4b0c-bbe1-498c60efc3b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:20.891616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d472eafe-b836-4b0c-bbe1-498c60efc3b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:20.892309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d472eafe-b836-4b0c-bbe1-498c60efc3b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:20.894023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d472eafe-b836-4b0c-bbe1-498c60efc3b7 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:20.905989 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1298/2585] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:20 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:09:20.909567 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0251a0be-ad0a-405c-87f8-115a51f17f44 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:20.934385 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0251a0be-ad0a-405c-87f8-115a51f17f44 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:20.941373 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d32d520-901d-43e8-bdf4-2aba828c1d89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:20.942096 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d32d520-901d-43e8-bdf4-2aba828c1d89 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:20.942428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d32d520-901d-43e8-bdf4-2aba828c1d89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:20.942698 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d32d520-901d-43e8-bdf4-2aba828c1d89 admin admin] PATH_INFO: `/v3/projects/288a1a257fa148a1b79a3bf3e7a77478/users/91bca22400ca4a95a9de66eb60051175/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:20.948007 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d32d520-901d-43e8-bdf4-2aba828c1d89 admin admin] RBAC: Authorizing `identity:create_grant(project_id=288a1a257fa148a1b79a3bf3e7a77478, user_id=91bca22400ca4a95a9de66eb60051175, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:20.948007 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d32d520-901d-43e8-bdf4-2aba828c1d89 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:20.969597 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1288/2586] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:09:20 2022] PUT /identity/v3/projects/288a1a257fa148a1b79a3bf3e7a77478/users/91bca22400ca4a95a9de66eb60051175/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:20.971595 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200e648e-d96b-45b5-b53f-3fbae3ce1bd5 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:20.974926 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200e648e-d96b-45b5-b53f-3fbae3ce1bd5 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:20.974926 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200e648e-d96b-45b5-b53f-3fbae3ce1bd5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:21.005751 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-200e648e-d96b-45b5-b53f-3fbae3ce1bd5 None None] MFA Rules not processed for user `91bca22400ca4a95a9de66eb60051175`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:21.007052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-200e648e-d96b-45b5-b53f-3fbae3ce1bd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:21.054703 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:21.092637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1299/2587] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:20 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:09:21.108225 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d32d520-901d-43e8-bdf4-2aba828c1d89 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:21.109560 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9d32d520-901d-43e8-bdf4-2aba828c1d89 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:21.242851 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d32d520-901d-43e8-bdf4-2aba828c1d89 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:21.248428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2865bc0-a926-49fe-89b2-ebdc0f4b7f58 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:21.254454 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2865bc0-a926-49fe-89b2-ebdc0f4b7f58 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:21.254454 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2865bc0-a926-49fe-89b2-ebdc0f4b7f58 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:21.254454 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2865bc0-a926-49fe-89b2-ebdc0f4b7f58 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:21.268789 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2865bc0-a926-49fe-89b2-ebdc0f4b7f58 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:21.270398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2865bc0-a926-49fe-89b2-ebdc0f4b7f58 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:21.288353 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1289/2588] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:21 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:21.785206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c780d4f-f97d-485f-8f50-128e4f0e3977 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:21.786454 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c780d4f-f97d-485f-8f50-128e4f0e3977 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:21.786949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c780d4f-f97d-485f-8f50-128e4f0e3977 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:21.809509 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-9c780d4f-f97d-485f-8f50-128e4f0e3977 None None] MFA Rules not processed for user `9359e90c4861470d9cc060abb9c3b203`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:21.810796 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-9c780d4f-f97d-485f-8f50-128e4f0e3977 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:21.874094 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1300/2589] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:21 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:22.646057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2865bc0-a926-49fe-89b2-ebdc0f4b7f58 service cinder] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:22.648411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2865bc0-a926-49fe-89b2-ebdc0f4b7f58 service cinder] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:22.653114 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb464004-38e9-487a-84a3-e8a47bae3231 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91bca22400ca4a95a9de66eb60051175', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '288a1a257fa148a1b79a3bf3e7a77478', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:22.654265 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb464004-38e9-487a-84a3-e8a47bae3231 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:22.654265 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb464004-38e9-487a-84a3-e8a47bae3231 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:22.654512 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb464004-38e9-487a-84a3-e8a47bae3231 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:22.655005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb464004-38e9-487a-84a3-e8a47bae3231 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:09:22.655320 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb464004-38e9-487a-84a3-e8a47bae3231 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:22.657055 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb464004-38e9-487a-84a3-e8a47bae3231 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:22.669872 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1290/2590] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:09:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:09:22.674387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c780d4f-f97d-485f-8f50-128e4f0e3977 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:22.686488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c780d4f-f97d-485f-8f50-128e4f0e3977 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:22.691328 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33c5df1-d9a5-43e2-8c1b-718f921a6da2 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91bca22400ca4a95a9de66eb60051175', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '288a1a257fa148a1b79a3bf3e7a77478', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:22.691982 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33c5df1-d9a5-43e2-8c1b-718f921a6da2 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:22.692487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33c5df1-d9a5-43e2-8c1b-718f921a6da2 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:22.692826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33c5df1-d9a5-43e2-8c1b-718f921a6da2 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:22.693527 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33c5df1-d9a5-43e2-8c1b-718f921a6da2 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:22.695541 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33c5df1-d9a5-43e2-8c1b-718f921a6da2 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:22.812107 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1301/2591] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:09:22 2022] POST /identity/v3/projects => generated 353 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:09:22.816148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb464004-38e9-487a-84a3-e8a47bae3231 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:22.826162 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb464004-38e9-487a-84a3-e8a47bae3231 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:22.833803 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:22.881819 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10061a3d-f1dd-403a-b269-00e9444e4298 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91bca22400ca4a95a9de66eb60051175', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '288a1a257fa148a1b79a3bf3e7a77478', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:22.882456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10061a3d-f1dd-403a-b269-00e9444e4298 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:22.882738 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10061a3d-f1dd-403a-b269-00e9444e4298 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:22.883007 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10061a3d-f1dd-403a-b269-00e9444e4298 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:22.883649 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10061a3d-f1dd-403a-b269-00e9444e4298 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:09:22.883987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10061a3d-f1dd-403a-b269-00e9444e4298 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:22.885589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10061a3d-f1dd-403a-b269-00e9444e4298 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:22.897455 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1291/2592] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:09:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:09:22.936064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33c5df1-d9a5-43e2-8c1b-718f921a6da2 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:22.946860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33c5df1-d9a5-43e2-8c1b-718f921a6da2 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:22.951439 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8800606-5ece-4c3d-b073-72170c5fa90a tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '91bca22400ca4a95a9de66eb60051175', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '288a1a257fa148a1b79a3bf3e7a77478', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:22.952057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8800606-5ece-4c3d-b073-72170c5fa90a tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:22.952335 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8800606-5ece-4c3d-b073-72170c5fa90a tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:22.952602 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8800606-5ece-4c3d-b073-72170c5fa90a tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] PATH_INFO: `/v3/projects/cc36db8334a5486b84b2dacdc29c3b0b` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:22.953718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8800606-5ece-4c3d-b073-72170c5fa90a tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] RBAC: Authorizing `identity:delete_project(project_id=cc36db8334a5486b84b2dacdc29c3b0b)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:22.955802 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8800606-5ece-4c3d-b073-72170c5fa90a tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:23.069761 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f8800606-5ece-4c3d-b073-72170c5fa90a tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cc36db8334a5486b84b2dacdc29c3b0b', 'request_id': 'req-f8800606-5ece-4c3d-b073-72170c5fa90a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:23.147012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f8800606-5ece-4c3d-b073-72170c5fa90a tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cc36db8334a5486b84b2dacdc29c3b0b', 'request_id': 'req-f8800606-5ece-4c3d-b073-72170c5fa90a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:23.160460 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f8800606-5ece-4c3d-b073-72170c5fa90a tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cc36db8334a5486b84b2dacdc29c3b0b', 'request_id': 'req-f8800606-5ece-4c3d-b073-72170c5fa90a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:23.178567 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f8800606-5ece-4c3d-b073-72170c5fa90a tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] The token cache is being invalidate because project cc36db8334a5486b84b2dacdc29c3b0b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:09:23.179204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f8800606-5ece-4c3d-b073-72170c5fa90a tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:23.180126 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f8800606-5ece-4c3d-b073-72170c5fa90a tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:23.182643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1302/2593] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:09:22 2022] DELETE /identity/v3/projects/cc36db8334a5486b84b2dacdc29c3b0b => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:23.994740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10061a3d-f1dd-403a-b269-00e9444e4298 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:23.996733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-10061a3d-f1dd-403a-b269-00e9444e4298 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:24.068836 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10061a3d-f1dd-403a-b269-00e9444e4298 tempest-VolumeQuotaClassesTest-421725783 tempest-VolumeQuotaClassesTest-421725783-project-admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:24.132895 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c5a7a0-4dee-48a3-9298-cf5089621f96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:24.133531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c5a7a0-4dee-48a3-9298-cf5089621f96 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:24.133898 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c5a7a0-4dee-48a3-9298-cf5089621f96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:24.134172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c5a7a0-4dee-48a3-9298-cf5089621f96 admin admin] PATH_INFO: `/v3/users/72eb449f379b45c89074100e07c2ac1a` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:24.135379 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c5a7a0-4dee-48a3-9298-cf5089621f96 admin admin] RBAC: Authorizing `identity:delete_user(user_id=72eb449f379b45c89074100e07c2ac1a)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:24.137433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c5a7a0-4dee-48a3-9298-cf5089621f96 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:24.339989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f1c5a7a0-4dee-48a3-9298-cf5089621f96 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '72eb449f379b45c89074100e07c2ac1a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:24.449908 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f1c5a7a0-4dee-48a3-9298-cf5089621f96 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '72eb449f379b45c89074100e07c2ac1a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:24.450667 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f1c5a7a0-4dee-48a3-9298-cf5089621f96 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '72eb449f379b45c89074100e07c2ac1a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:24.452777 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:24.464457 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f1c5a7a0-4dee-48a3-9298-cf5089621f96 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '72eb449f379b45c89074100e07c2ac1a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:24.473242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f1c5a7a0-4dee-48a3-9298-cf5089621f96 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '72eb449f379b45c89074100e07c2ac1a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:24.479751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1292/2594] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:09:23 2022] DELETE /identity/v3/users/72eb449f379b45c89074100e07c2ac1a => generated 0 bytes in 486 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:09:24.484923 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be681f5-5e24-41e5-a76f-7ad8852630e7 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:24.485427 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be681f5-5e24-41e5-a76f-7ad8852630e7 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:24.485551 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be681f5-5e24-41e5-a76f-7ad8852630e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:24.505779 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-2be681f5-5e24-41e5-a76f-7ad8852630e7 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:24.507024 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-2be681f5-5e24-41e5-a76f-7ad8852630e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:24.555166 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:24.570374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1303/2595] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:09:24.580617 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c5a7a0-4dee-48a3-9298-cf5089621f96 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:24.582657 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f1c5a7a0-4dee-48a3-9298-cf5089621f96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:24.646638 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c5a7a0-4dee-48a3-9298-cf5089621f96 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:24.654179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493220b7-7eb6-406e-aead-499474284830 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:24.654179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493220b7-7eb6-406e-aead-499474284830 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:24.654179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493220b7-7eb6-406e-aead-499474284830 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:24.654179 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493220b7-7eb6-406e-aead-499474284830 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:24.663208 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493220b7-7eb6-406e-aead-499474284830 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:24.664633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-493220b7-7eb6-406e-aead-499474284830 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:24.676209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1293/2596] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:09:25.221216 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be681f5-5e24-41e5-a76f-7ad8852630e7 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:25.222831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-2be681f5-5e24-41e5-a76f-7ad8852630e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:25.242831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be681f5-5e24-41e5-a76f-7ad8852630e7 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:25.249629 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29a94d3-8118-4601-ad12-b481776c2d81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:25.250284 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29a94d3-8118-4601-ad12-b481776c2d81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:25.250598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29a94d3-8118-4601-ad12-b481776c2d81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:25.250919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29a94d3-8118-4601-ad12-b481776c2d81 admin admin] PATH_INFO: `/v3/projects/3b85a316ecef43f7bb9769abc1e24adf` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:25.252041 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29a94d3-8118-4601-ad12-b481776c2d81 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3b85a316ecef43f7bb9769abc1e24adf)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:25.253977 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29a94d3-8118-4601-ad12-b481776c2d81 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:25.433553 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f29a94d3-8118-4601-ad12-b481776c2d81 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3b85a316ecef43f7bb9769abc1e24adf', 'request_id': 'req-f29a94d3-8118-4601-ad12-b481776c2d81'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:25.557425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f29a94d3-8118-4601-ad12-b481776c2d81 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3b85a316ecef43f7bb9769abc1e24adf', 'request_id': 'req-f29a94d3-8118-4601-ad12-b481776c2d81'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:25.565557 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f29a94d3-8118-4601-ad12-b481776c2d81 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3b85a316ecef43f7bb9769abc1e24adf', 'request_id': 'req-f29a94d3-8118-4601-ad12-b481776c2d81'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:25.583378 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f29a94d3-8118-4601-ad12-b481776c2d81 admin admin] The token cache is being invalidate because project 3b85a316ecef43f7bb9769abc1e24adf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:09:25.583378 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f29a94d3-8118-4601-ad12-b481776c2d81 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:25.583378 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-f29a94d3-8118-4601-ad12-b481776c2d81 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:25.583378 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1304/2597] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:09:25 2022] DELETE /identity/v3/projects/3b85a316ecef43f7bb9769abc1e24adf => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:25.586150 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493220b7-7eb6-406e-aead-499474284830 service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:25.602957 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493220b7-7eb6-406e-aead-499474284830 service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:25.663830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24f699c-8027-4622-bbfb-931477df5d80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:25.664461 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24f699c-8027-4622-bbfb-931477df5d80 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:25.664742 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24f699c-8027-4622-bbfb-931477df5d80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:25.665016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24f699c-8027-4622-bbfb-931477df5d80 admin admin] PATH_INFO: `/v3/users/91bca22400ca4a95a9de66eb60051175` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:25.666178 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24f699c-8027-4622-bbfb-931477df5d80 admin admin] RBAC: Authorizing `identity:delete_user(user_id=91bca22400ca4a95a9de66eb60051175)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:25.668298 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24f699c-8027-4622-bbfb-931477df5d80 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:25.902770 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-c24f699c-8027-4622-bbfb-931477df5d80 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '91bca22400ca4a95a9de66eb60051175'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:25.987254 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-c24f699c-8027-4622-bbfb-931477df5d80 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '91bca22400ca4a95a9de66eb60051175'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:25.987896 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-c24f699c-8027-4622-bbfb-931477df5d80 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '91bca22400ca4a95a9de66eb60051175'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:26.001853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-c24f699c-8027-4622-bbfb-931477df5d80 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '91bca22400ca4a95a9de66eb60051175'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:26.006426 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:26.009463 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-c24f699c-8027-4622-bbfb-931477df5d80 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '91bca22400ca4a95a9de66eb60051175'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:26.015254 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1294/2598] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:09:25 2022] DELETE /identity/v3/users/91bca22400ca4a95a9de66eb60051175 => generated 0 bytes in 430 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:26.434627 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29a94d3-8118-4601-ad12-b481776c2d81 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:26.436055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f29a94d3-8118-4601-ad12-b481776c2d81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:26.454633 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29a94d3-8118-4601-ad12-b481776c2d81 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:26.510950 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbd61dd-0afb-44a9-8d91-c6c140aa9dc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:26.511582 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbd61dd-0afb-44a9-8d91-c6c140aa9dc5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:26.511861 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbd61dd-0afb-44a9-8d91-c6c140aa9dc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:26.512137 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbd61dd-0afb-44a9-8d91-c6c140aa9dc5 admin admin] PATH_INFO: `/v3/projects/288a1a257fa148a1b79a3bf3e7a77478` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:26.513310 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbd61dd-0afb-44a9-8d91-c6c140aa9dc5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=288a1a257fa148a1b79a3bf3e7a77478)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:26.516013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbd61dd-0afb-44a9-8d91-c6c140aa9dc5 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:26.621653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6cbd61dd-0afb-44a9-8d91-c6c140aa9dc5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '288a1a257fa148a1b79a3bf3e7a77478', 'request_id': 'req-6cbd61dd-0afb-44a9-8d91-c6c140aa9dc5'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:26.690647 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6cbd61dd-0afb-44a9-8d91-c6c140aa9dc5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '288a1a257fa148a1b79a3bf3e7a77478', 'request_id': 'req-6cbd61dd-0afb-44a9-8d91-c6c140aa9dc5'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:26.698247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6cbd61dd-0afb-44a9-8d91-c6c140aa9dc5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '288a1a257fa148a1b79a3bf3e7a77478', 'request_id': 'req-6cbd61dd-0afb-44a9-8d91-c6c140aa9dc5'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:26.711606 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6cbd61dd-0afb-44a9-8d91-c6c140aa9dc5 admin admin] The token cache is being invalidate because project 288a1a257fa148a1b79a3bf3e7a77478 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:09:26.711916 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6cbd61dd-0afb-44a9-8d91-c6c140aa9dc5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:26.712677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-6cbd61dd-0afb-44a9-8d91-c6c140aa9dc5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:26.714155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1305/2599] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:09:26 2022] DELETE /identity/v3/projects/288a1a257fa148a1b79a3bf3e7a77478 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:26.719063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c868b23-4e87-4fbd-8d8e-6810feec3cb2 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:26.719892 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c868b23-4e87-4fbd-8d8e-6810feec3cb2 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:26.720279 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c868b23-4e87-4fbd-8d8e-6810feec3cb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:26.740280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-3c868b23-4e87-4fbd-8d8e-6810feec3cb2 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:26.741494 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3c868b23-4e87-4fbd-8d8e-6810feec3cb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:26.807665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1295/2600] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:26.816282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7733e61-6acd-49b3-adc9-7b60f1f805ef None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:26.816565 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7733e61-6acd-49b3-adc9-7b60f1f805ef None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:26.816839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7733e61-6acd-49b3-adc9-7b60f1f805ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:26.825108 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:26.838718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-d7733e61-6acd-49b3-adc9-7b60f1f805ef None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:26.839925 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d7733e61-6acd-49b3-adc9-7b60f1f805ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:26.848103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1306/2601] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:09:26.852519 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c868b23-4e87-4fbd-8d8e-6810feec3cb2 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:26.863125 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c868b23-4e87-4fbd-8d8e-6810feec3cb2 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:26.867870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775129ce-0a38-44ea-8b50-b4b948360875 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:26.868479 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775129ce-0a38-44ea-8b50-b4b948360875 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:26.868753 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775129ce-0a38-44ea-8b50-b4b948360875 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:26.869019 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775129ce-0a38-44ea-8b50-b4b948360875 admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:26.869666 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775129ce-0a38-44ea-8b50-b4b948360875 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:09:26.869964 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775129ce-0a38-44ea-8b50-b4b948360875 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:26.871755 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775129ce-0a38-44ea-8b50-b4b948360875 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:26.886229 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1296/2602] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:09:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:09:26.891193 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7733e61-6acd-49b3-adc9-7b60f1f805ef None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:26.900711 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7733e61-6acd-49b3-adc9-7b60f1f805ef None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:26.907613 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a924ef-7d7a-4992-aacd-d284e787d9a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:26.908345 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a924ef-7d7a-4992-aacd-d284e787d9a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:26.908631 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a924ef-7d7a-4992-aacd-d284e787d9a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:26.908969 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a924ef-7d7a-4992-aacd-d284e787d9a1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:26.909702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a924ef-7d7a-4992-aacd-d284e787d9a1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:26.912161 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a924ef-7d7a-4992-aacd-d284e787d9a1 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:27.845092 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1307/2603] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:09:26 2022] POST /identity/v3/projects => generated 379 bytes in 951 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:09:27.846210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775129ce-0a38-44ea-8b50-b4b948360875 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:27.857181 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775129ce-0a38-44ea-8b50-b4b948360875 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:27.932531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2e3696-e7fb-4801-827f-a1b1834361c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:27.933148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2e3696-e7fb-4801-827f-a1b1834361c0 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:27.933427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2e3696-e7fb-4801-827f-a1b1834361c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:27.933700 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2e3696-e7fb-4801-827f-a1b1834361c0 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:27.934359 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2e3696-e7fb-4801-827f-a1b1834361c0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:27.936391 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2e3696-e7fb-4801-827f-a1b1834361c0 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:28.097522 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1297/2604] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:09:27 2022] POST /identity/v3/users => generated 290 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:09:28.101457 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a924ef-7d7a-4992-aacd-d284e787d9a1 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:28.112752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a924ef-7d7a-4992-aacd-d284e787d9a1 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:28.117080 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26bf4729-88f3-403d-ae3d-84ae0804c422 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:28.117714 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bf4729-88f3-403d-ae3d-84ae0804c422 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:28.117990 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bf4729-88f3-403d-ae3d-84ae0804c422 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:28.118257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bf4729-88f3-403d-ae3d-84ae0804c422 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:28.118931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26bf4729-88f3-403d-ae3d-84ae0804c422 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:28.120542 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26bf4729-88f3-403d-ae3d-84ae0804c422 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:28.131362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1308/2605] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:09:28.135472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2e3696-e7fb-4801-827f-a1b1834361c0 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:28.146830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2e3696-e7fb-4801-827f-a1b1834361c0 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:28.151587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aba6f70-3b90-4dcb-9d3b-95bdfc36f049 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:28.152218 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aba6f70-3b90-4dcb-9d3b-95bdfc36f049 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:28.156599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aba6f70-3b90-4dcb-9d3b-95bdfc36f049 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:28.156599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aba6f70-3b90-4dcb-9d3b-95bdfc36f049 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:28.156599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aba6f70-3b90-4dcb-9d3b-95bdfc36f049 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:28.156599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aba6f70-3b90-4dcb-9d3b-95bdfc36f049 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:28.167139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1298/2606] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:28 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:09:28.171632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26bf4729-88f3-403d-ae3d-84ae0804c422 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:28.181682 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26bf4729-88f3-403d-ae3d-84ae0804c422 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:28.187493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9134802f-b23f-4f9d-affa-45825d68d4c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:28.188153 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9134802f-b23f-4f9d-affa-45825d68d4c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:28.188431 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9134802f-b23f-4f9d-affa-45825d68d4c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:28.188700 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9134802f-b23f-4f9d-affa-45825d68d4c2 admin admin] PATH_INFO: `/v3/projects/43ce2061e19f498ca3bf5a9fb7dc4a6a/users/74c954e7dfbd4eb5bbf76aeec1785425/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:28.206294 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9134802f-b23f-4f9d-affa-45825d68d4c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=43ce2061e19f498ca3bf5a9fb7dc4a6a, user_id=74c954e7dfbd4eb5bbf76aeec1785425, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:28.209280 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9134802f-b23f-4f9d-affa-45825d68d4c2 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:28.270884 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aba6f70-3b90-4dcb-9d3b-95bdfc36f049 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:28.273193 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-0aba6f70-3b90-4dcb-9d3b-95bdfc36f049 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:28.314288 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1309/2607] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:09:28 2022] PUT /identity/v3/projects/43ce2061e19f498ca3bf5a9fb7dc4a6a/users/74c954e7dfbd4eb5bbf76aeec1785425/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:28.317342 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bd1d2d-6711-47c4-bd82-a2338b8de7d8 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:28.317631 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bd1d2d-6711-47c4-bd82-a2338b8de7d8 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:28.317902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bd1d2d-6711-47c4-bd82-a2338b8de7d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:28.351685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-09bd1d2d-6711-47c4-bd82-a2338b8de7d8 None None] MFA Rules not processed for user `74c954e7dfbd4eb5bbf76aeec1785425`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:28.352836 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-09bd1d2d-6711-47c4-bd82-a2338b8de7d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:28.354161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:28.363786 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aba6f70-3b90-4dcb-9d3b-95bdfc36f049 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:28.423670 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1310/2608] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:28 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:28.427967 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fda3721-c147-4e8e-9092-9902d6697a3b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce249d4a99c34891b6ba941f57a14dd4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:28.428567 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fda3721-c147-4e8e-9092-9902d6697a3b service placement] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:28.428849 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fda3721-c147-4e8e-9092-9902d6697a3b service placement] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:28.429219 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fda3721-c147-4e8e-9092-9902d6697a3b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:28.431106 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9fda3721-c147-4e8e-9092-9902d6697a3b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:28.449654 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09bd1d2d-6711-47c4-bd82-a2338b8de7d8 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:28.449654 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-09bd1d2d-6711-47c4-bd82-a2338b8de7d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:28.449863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fda3721-c147-4e8e-9092-9902d6697a3b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:28.449863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fda3721-c147-4e8e-9092-9902d6697a3b service placement] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:28.459407 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1299/2609] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:28 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:09:28.509088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09bd1d2d-6711-47c4-bd82-a2338b8de7d8 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:28.515917 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67265f7-267d-474c-8755-dd54f8ce0c09 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:28.516573 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67265f7-267d-474c-8755-dd54f8ce0c09 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:28.516875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67265f7-267d-474c-8755-dd54f8ce0c09 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:28.517179 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67265f7-267d-474c-8755-dd54f8ce0c09 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:28.526119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67265f7-267d-474c-8755-dd54f8ce0c09 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:28.527585 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67265f7-267d-474c-8755-dd54f8ce0c09 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:28.538169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1311/2610] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:28 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:28.558886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fda3721-c147-4e8e-9092-9902d6697a3b service placement] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:28.648558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fda3721-c147-4e8e-9092-9902d6697a3b service placement] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:28.653610 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ee60b38-155c-4f11-80dd-63c66757d4ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:28.654297 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee60b38-155c-4f11-80dd-63c66757d4ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:28.654598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee60b38-155c-4f11-80dd-63c66757d4ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:28.654866 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee60b38-155c-4f11-80dd-63c66757d4ca admin admin] PATH_INFO: `/v3/users/74c954e7dfbd4eb5bbf76aeec1785425` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:28.656576 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee60b38-155c-4f11-80dd-63c66757d4ca admin admin] RBAC: Authorizing `identity:delete_user(user_id=74c954e7dfbd4eb5bbf76aeec1785425)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:28.658555 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ee60b38-155c-4f11-80dd-63c66757d4ca admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:28.921139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1ee60b38-155c-4f11-80dd-63c66757d4ca admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '74c954e7dfbd4eb5bbf76aeec1785425'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:29.028433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1ee60b38-155c-4f11-80dd-63c66757d4ca admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '74c954e7dfbd4eb5bbf76aeec1785425'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:29.029309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1ee60b38-155c-4f11-80dd-63c66757d4ca admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '74c954e7dfbd4eb5bbf76aeec1785425'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:29.046227 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1ee60b38-155c-4f11-80dd-63c66757d4ca admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '74c954e7dfbd4eb5bbf76aeec1785425'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:29.053214 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1ee60b38-155c-4f11-80dd-63c66757d4ca admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '74c954e7dfbd4eb5bbf76aeec1785425'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:29.067255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1300/2611] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:09:28 2022] DELETE /identity/v3/users/74c954e7dfbd4eb5bbf76aeec1785425 => generated 0 bytes in 509 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:29.071671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abd226f-e057-47ea-b67e-fad00ee85beb None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:29.072189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abd226f-e057-47ea-b67e-fad00ee85beb None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:29.072189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abd226f-e057-47ea-b67e-fad00ee85beb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:29.102414 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66475f9d-f78c-4ff9-9c53-4322df640084 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:29.102859 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66475f9d-f78c-4ff9-9c53-4322df640084 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:29.103303 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66475f9d-f78c-4ff9-9c53-4322df640084 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:29.104741 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-3abd226f-e057-47ea-b67e-fad00ee85beb None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:29.106252 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-3abd226f-e057-47ea-b67e-fad00ee85beb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:29.127064 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-66475f9d-f78c-4ff9-9c53-4322df640084 None None] MFA Rules not processed for user `b471d768cd814ff8b4f6bd35d21e24a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:29.128361 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-66475f9d-f78c-4ff9-9c53-4322df640084 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:29.147381 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:29.184084 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:29.191573 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1312/2612] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:29.206494 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1301/2613] 10.222.0.62 () {58 vars in 967 bytes} [Fri Sep 23 14:09:29 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:09:29.207409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abd226f-e057-47ea-b67e-fad00ee85beb None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:29.209530 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-3abd226f-e057-47ea-b67e-fad00ee85beb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:29.220133 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66475f9d-f78c-4ff9-9c53-4322df640084 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:29.221685 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-66475f9d-f78c-4ff9-9c53-4322df640084 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:29.296272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abd226f-e057-47ea-b67e-fad00ee85beb None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:29.312095 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7442c8-fbfa-4db5-8c8a-81eacf2f988f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:29.312780 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7442c8-fbfa-4db5-8c8a-81eacf2f988f service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:29.312978 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7442c8-fbfa-4db5-8c8a-81eacf2f988f service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:29.313243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7442c8-fbfa-4db5-8c8a-81eacf2f988f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:29.316849 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66475f9d-f78c-4ff9-9c53-4322df640084 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:29.322990 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99846a8d-4625-4730-8a52-e3b2a1bdf21a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b9269509f1d454eab7a3008c2ba0039', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:29.323874 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99846a8d-4625-4730-8a52-e3b2a1bdf21a service swift] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:29.324148 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99846a8d-4625-4730-8a52-e3b2a1bdf21a service swift] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:29.324405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7442c8-fbfa-4db5-8c8a-81eacf2f988f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:29.324511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99846a8d-4625-4730-8a52-e3b2a1bdf21a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:29.326371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7442c8-fbfa-4db5-8c8a-81eacf2f988f service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:29.334802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99846a8d-4625-4730-8a52-e3b2a1bdf21a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:29.336433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99846a8d-4625-4730-8a52-e3b2a1bdf21a service swift] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:29.337175 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1313/2614] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:29.337967 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7442c8-fbfa-4db5-8c8a-81eacf2f988f service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:29.339785 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8d7442c8-fbfa-4db5-8c8a-81eacf2f988f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:29.346809 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1302/2615] 10.222.0.62 () {60 vars in 1384 bytes} [Fri Sep 23 14:09:29 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 23 14:09:29.354528 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7442c8-fbfa-4db5-8c8a-81eacf2f988f service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:29.364804 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802162f7-4d41-465e-acb8-714325c2237f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:29.365452 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802162f7-4d41-465e-acb8-714325c2237f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:29.365727 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802162f7-4d41-465e-acb8-714325c2237f admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:29.366017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802162f7-4d41-465e-acb8-714325c2237f admin admin] PATH_INFO: `/v3/users/2af73db67a574429b59df945eccf74af` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:29.367174 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802162f7-4d41-465e-acb8-714325c2237f admin admin] RBAC: Authorizing `identity:delete_user(user_id=2af73db67a574429b59df945eccf74af)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:29.372498 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802162f7-4d41-465e-acb8-714325c2237f admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:29.621119 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-802162f7-4d41-465e-acb8-714325c2237f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2af73db67a574429b59df945eccf74af'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:29.690791 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-802162f7-4d41-465e-acb8-714325c2237f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2af73db67a574429b59df945eccf74af'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:29.704924 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-802162f7-4d41-465e-acb8-714325c2237f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2af73db67a574429b59df945eccf74af'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:29.706022 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-802162f7-4d41-465e-acb8-714325c2237f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2af73db67a574429b59df945eccf74af'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:29.714615 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-802162f7-4d41-465e-acb8-714325c2237f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2af73db67a574429b59df945eccf74af'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:29.722414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1314/2616] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:09:29 2022] DELETE /identity/v3/users/2af73db67a574429b59df945eccf74af => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:29.921789 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99846a8d-4625-4730-8a52-e3b2a1bdf21a service swift] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:29.924995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-99846a8d-4625-4730-8a52-e3b2a1bdf21a service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:29.941682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99846a8d-4625-4730-8a52-e3b2a1bdf21a service swift] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:30.019080 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802162f7-4d41-465e-acb8-714325c2237f admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:30.021830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-802162f7-4d41-465e-acb8-714325c2237f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:30.064497 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c7567f-051b-41ce-9772-37fd6d92321a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:30.065238 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c7567f-051b-41ce-9772-37fd6d92321a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:30.065532 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c7567f-051b-41ce-9772-37fd6d92321a admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:30.065811 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c7567f-051b-41ce-9772-37fd6d92321a admin admin] PATH_INFO: `/v3/projects/43ce2061e19f498ca3bf5a9fb7dc4a6a` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:30.071395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c7567f-051b-41ce-9772-37fd6d92321a admin admin] RBAC: Authorizing `identity:delete_project(project_id=43ce2061e19f498ca3bf5a9fb7dc4a6a)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:30.072311 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c7567f-051b-41ce-9772-37fd6d92321a admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:30.127181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802162f7-4d41-465e-acb8-714325c2237f admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:30.131709 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891a0383-e101-4695-b1bc-57ec7a5f70fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:30.132329 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891a0383-e101-4695-b1bc-57ec7a5f70fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:30.132604 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891a0383-e101-4695-b1bc-57ec7a5f70fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:30.132868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891a0383-e101-4695-b1bc-57ec7a5f70fe admin admin] PATH_INFO: `/v3/projects/3ef0c5d7373640d6bb149b011f074ae3` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:30.133974 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891a0383-e101-4695-b1bc-57ec7a5f70fe admin admin] RBAC: Authorizing `identity:delete_project(project_id=3ef0c5d7373640d6bb149b011f074ae3)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:30.135886 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-891a0383-e101-4695-b1bc-57ec7a5f70fe admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:30.281936 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f4c7567f-051b-41ce-9772-37fd6d92321a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '43ce2061e19f498ca3bf5a9fb7dc4a6a', 'request_id': 'req-f4c7567f-051b-41ce-9772-37fd6d92321a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:30.285144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-891a0383-e101-4695-b1bc-57ec7a5f70fe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3ef0c5d7373640d6bb149b011f074ae3', 'request_id': 'req-891a0383-e101-4695-b1bc-57ec7a5f70fe'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:30.313905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:30.365038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f4c7567f-051b-41ce-9772-37fd6d92321a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '43ce2061e19f498ca3bf5a9fb7dc4a6a', 'request_id': 'req-f4c7567f-051b-41ce-9772-37fd6d92321a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:30.373253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f4c7567f-051b-41ce-9772-37fd6d92321a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '43ce2061e19f498ca3bf5a9fb7dc4a6a', 'request_id': 'req-f4c7567f-051b-41ce-9772-37fd6d92321a'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:30.385959 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f4c7567f-051b-41ce-9772-37fd6d92321a admin admin] The token cache is being invalidate because project 43ce2061e19f498ca3bf5a9fb7dc4a6a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:09:30.386353 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f4c7567f-051b-41ce-9772-37fd6d92321a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:30.387236 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f4c7567f-051b-41ce-9772-37fd6d92321a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:30.389068 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1303/2617] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:09:29 2022] DELETE /identity/v3/projects/43ce2061e19f498ca3bf5a9fb7dc4a6a => generated 0 bytes in 468 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:30.394319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb38ce87-8ea3-4a66-a749-49ddde73f691 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:30.394671 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb38ce87-8ea3-4a66-a749-49ddde73f691 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:30.394951 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb38ce87-8ea3-4a66-a749-49ddde73f691 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:30.418176 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-eb38ce87-8ea3-4a66-a749-49ddde73f691 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:30.419615 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-eb38ce87-8ea3-4a66-a749-49ddde73f691 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:30.421618 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-891a0383-e101-4695-b1bc-57ec7a5f70fe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3ef0c5d7373640d6bb149b011f074ae3', 'request_id': 'req-891a0383-e101-4695-b1bc-57ec7a5f70fe'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:30.428077 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-891a0383-e101-4695-b1bc-57ec7a5f70fe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3ef0c5d7373640d6bb149b011f074ae3', 'request_id': 'req-891a0383-e101-4695-b1bc-57ec7a5f70fe'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:30.441380 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-891a0383-e101-4695-b1bc-57ec7a5f70fe admin admin] The token cache is being invalidate because project 3ef0c5d7373640d6bb149b011f074ae3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:09:30.441904 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-891a0383-e101-4695-b1bc-57ec7a5f70fe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:30.442884 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-891a0383-e101-4695-b1bc-57ec7a5f70fe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:30.444423 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1315/2618] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:09:30 2022] DELETE /identity/v3/projects/3ef0c5d7373640d6bb149b011f074ae3 => generated 0 bytes in 426 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:30.449647 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891a0383-e101-4695-b1bc-57ec7a5f70fe admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:30.453275 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-891a0383-e101-4695-b1bc-57ec7a5f70fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:30.532727 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1304/2619] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:30.536259 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891a0383-e101-4695-b1bc-57ec7a5f70fe admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:30.540287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54efa5b-728f-492c-8bb6-d5ff2b5f8833 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:30.540758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54efa5b-728f-492c-8bb6-d5ff2b5f8833 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:30.541033 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54efa5b-728f-492c-8bb6-d5ff2b5f8833 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:30.544732 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb1ed1b-9b75-4440-b513-aa59322ebe70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:30.546456 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb1ed1b-9b75-4440-b513-aa59322ebe70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:30.547287 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb1ed1b-9b75-4440-b513-aa59322ebe70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:30.547769 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb1ed1b-9b75-4440-b513-aa59322ebe70 admin admin] PATH_INFO: `/v3/users/5c606509ef13421eafd1e54d556b9c8e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:30.550505 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb1ed1b-9b75-4440-b513-aa59322ebe70 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5c606509ef13421eafd1e54d556b9c8e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:30.552737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb1ed1b-9b75-4440-b513-aa59322ebe70 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:30.570733 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-a54efa5b-728f-492c-8bb6-d5ff2b5f8833 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:30.572110 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a54efa5b-728f-492c-8bb6-d5ff2b5f8833 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:30.587618 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1305/2620] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:30.589096 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54efa5b-728f-492c-8bb6-d5ff2b5f8833 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:30.600529 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54efa5b-728f-492c-8bb6-d5ff2b5f8833 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:30.605329 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039d0cab-ce91-498b-9e18-5fe166587aaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:30.605934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039d0cab-ce91-498b-9e18-5fe166587aaa admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:30.606207 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039d0cab-ce91-498b-9e18-5fe166587aaa admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:30.606497 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039d0cab-ce91-498b-9e18-5fe166587aaa admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:30.607396 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039d0cab-ce91-498b-9e18-5fe166587aaa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:09:30.607751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039d0cab-ce91-498b-9e18-5fe166587aaa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:30.609792 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039d0cab-ce91-498b-9e18-5fe166587aaa admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:30.622208 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1306/2621] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:09:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:09:30.627202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039d0cab-ce91-498b-9e18-5fe166587aaa admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:30.636984 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039d0cab-ce91-498b-9e18-5fe166587aaa admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:30.642812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c2bec2-2b20-4295-9d18-576cce149cf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:30.648528 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c2bec2-2b20-4295-9d18-576cce149cf6 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:30.648528 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c2bec2-2b20-4295-9d18-576cce149cf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:30.648528 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c2bec2-2b20-4295-9d18-576cce149cf6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:30.648528 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c2bec2-2b20-4295-9d18-576cce149cf6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:30.648528 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c2bec2-2b20-4295-9d18-576cce149cf6 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:30.754694 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1307/2622] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:09:30 2022] POST /identity/v3/projects => generated 383 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:09:30.758761 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c2bec2-2b20-4295-9d18-576cce149cf6 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:30.820399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ecb1ed1b-9b75-4440-b513-aa59322ebe70 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5c606509ef13421eafd1e54d556b9c8e'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:30.825043 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c2bec2-2b20-4295-9d18-576cce149cf6 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:30.830256 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892dbdfa-96cd-48ba-85d2-0f15c24f2ebb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:30.830893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892dbdfa-96cd-48ba-85d2-0f15c24f2ebb admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:30.831173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892dbdfa-96cd-48ba-85d2-0f15c24f2ebb admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:30.831441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892dbdfa-96cd-48ba-85d2-0f15c24f2ebb admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:30.832068 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892dbdfa-96cd-48ba-85d2-0f15c24f2ebb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:30.833968 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892dbdfa-96cd-48ba-85d2-0f15c24f2ebb admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:30.930165 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ecb1ed1b-9b75-4440-b513-aa59322ebe70 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5c606509ef13421eafd1e54d556b9c8e'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:30.932279 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:30.940218 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1308/2623] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:09:30 2022] POST /identity/v3/users => generated 292 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:09:30.943834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892dbdfa-96cd-48ba-85d2-0f15c24f2ebb admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:30.948275 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ecb1ed1b-9b75-4440-b513-aa59322ebe70 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5c606509ef13421eafd1e54d556b9c8e'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:30.949887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ecb1ed1b-9b75-4440-b513-aa59322ebe70 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5c606509ef13421eafd1e54d556b9c8e'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:30.956456 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892dbdfa-96cd-48ba-85d2-0f15c24f2ebb admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:30.959976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ecb1ed1b-9b75-4440-b513-aa59322ebe70 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5c606509ef13421eafd1e54d556b9c8e'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:30.964363 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e1b8e5-e465-4d82-b377-762f9a28edd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:30.964363 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e1b8e5-e465-4d82-b377-762f9a28edd6 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:30.964363 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e1b8e5-e465-4d82-b377-762f9a28edd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:30.964363 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e1b8e5-e465-4d82-b377-762f9a28edd6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:30.964790 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e1b8e5-e465-4d82-b377-762f9a28edd6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:30.966443 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1316/2624] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:09:30 2022] DELETE /identity/v3/users/5c606509ef13421eafd1e54d556b9c8e => generated 0 bytes in 518 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:30.966695 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e1b8e5-e465-4d82-b377-762f9a28edd6 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:30.977268 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1309/2625] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:30 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:09:30.981210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb1ed1b-9b75-4440-b513-aa59322ebe70 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:30.982700 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-ecb1ed1b-9b75-4440-b513-aa59322ebe70 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:31.087057 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb1ed1b-9b75-4440-b513-aa59322ebe70 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:31.095880 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf70b9a-04b8-4ea4-90bb-13100f37967b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:31.096891 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf70b9a-04b8-4ea4-90bb-13100f37967b admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:31.097365 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf70b9a-04b8-4ea4-90bb-13100f37967b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:31.097845 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf70b9a-04b8-4ea4-90bb-13100f37967b admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:31.099070 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf70b9a-04b8-4ea4-90bb-13100f37967b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:31.102080 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf70b9a-04b8-4ea4-90bb-13100f37967b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:31.124850 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1317/2626] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:30 2022] GET /identity/v3/roles => generated 1420 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:09:31.127669 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e1b8e5-e465-4d82-b377-762f9a28edd6 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:31.139002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e1b8e5-e465-4d82-b377-762f9a28edd6 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:31.144553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a81f727-3620-490f-9b59-2caf118ee0ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:31.147890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a81f727-3620-490f-9b59-2caf118ee0ec admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:31.147890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a81f727-3620-490f-9b59-2caf118ee0ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:31.147890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a81f727-3620-490f-9b59-2caf118ee0ec admin admin] PATH_INFO: `/v3/projects/da6f07b1dac849288b096121811d770a/users/c422c0fc22e9411e92f1945583082308/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:31.163124 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a81f727-3620-490f-9b59-2caf118ee0ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=da6f07b1dac849288b096121811d770a, user_id=c422c0fc22e9411e92f1945583082308, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:31.166874 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a81f727-3620-490f-9b59-2caf118ee0ec admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:31.257728 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1310/2627] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:09:31 2022] PUT /identity/v3/projects/da6f07b1dac849288b096121811d770a/users/c422c0fc22e9411e92f1945583082308/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:31.268685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674f6d7c-6dc4-4d56-b3da-2678be05092a None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:31.269183 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674f6d7c-6dc4-4d56-b3da-2678be05092a None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:31.269306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674f6d7c-6dc4-4d56-b3da-2678be05092a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:31.297791 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-674f6d7c-6dc4-4d56-b3da-2678be05092a None None] MFA Rules not processed for user `c422c0fc22e9411e92f1945583082308`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:31.299046 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-674f6d7c-6dc4-4d56-b3da-2678be05092a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:31.337354 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a81f727-3620-490f-9b59-2caf118ee0ec admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:31.339374 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-7a81f727-3620-490f-9b59-2caf118ee0ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:31.378469 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1318/2628] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:31 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:31.392215 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674f6d7c-6dc4-4d56-b3da-2678be05092a None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:31.393622 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-674f6d7c-6dc4-4d56-b3da-2678be05092a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:31.409531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a81f727-3620-490f-9b59-2caf118ee0ec admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:31.414579 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1049dee8-f64e-4bab-a989-ecdb72629298 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:31.415422 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1049dee8-f64e-4bab-a989-ecdb72629298 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:31.415870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1049dee8-f64e-4bab-a989-ecdb72629298 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:31.416170 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1049dee8-f64e-4bab-a989-ecdb72629298 admin admin] PATH_INFO: `/v3/projects/e001a2e371f7483ab56127aca94971fc` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:31.417330 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1049dee8-f64e-4bab-a989-ecdb72629298 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e001a2e371f7483ab56127aca94971fc)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:31.419280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1049dee8-f64e-4bab-a989-ecdb72629298 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:31.445020 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:31.454708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674f6d7c-6dc4-4d56-b3da-2678be05092a None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:31.458370 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be1f6a48-00a4-45a1-a3a6-b8a099fbcaab service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:31.459022 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1f6a48-00a4-45a1-a3a6-b8a099fbcaab service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:31.459301 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1f6a48-00a4-45a1-a3a6-b8a099fbcaab service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:31.459572 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be1f6a48-00a4-45a1-a3a6-b8a099fbcaab service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:31.467280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:31.468414 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1f6a48-00a4-45a1-a3a6-b8a099fbcaab service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:31.469840 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be1f6a48-00a4-45a1-a3a6-b8a099fbcaab service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:31.478942 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1319/2629] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:31 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:31.519964 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1049dee8-f64e-4bab-a989-ecdb72629298 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e001a2e371f7483ab56127aca94971fc', 'request_id': 'req-1049dee8-f64e-4bab-a989-ecdb72629298'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:31.585592 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1049dee8-f64e-4bab-a989-ecdb72629298 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e001a2e371f7483ab56127aca94971fc', 'request_id': 'req-1049dee8-f64e-4bab-a989-ecdb72629298'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:31.591622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1049dee8-f64e-4bab-a989-ecdb72629298 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e001a2e371f7483ab56127aca94971fc', 'request_id': 'req-1049dee8-f64e-4bab-a989-ecdb72629298'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:31.604312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1049dee8-f64e-4bab-a989-ecdb72629298 admin admin] The token cache is being invalidate because project e001a2e371f7483ab56127aca94971fc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:09:31.604811 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1049dee8-f64e-4bab-a989-ecdb72629298 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:31.605504 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-1049dee8-f64e-4bab-a989-ecdb72629298 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:31.607205 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1311/2630] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:09:31 2022] DELETE /identity/v3/projects/e001a2e371f7483ab56127aca94971fc => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:09:31.612893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52fb374-e283-48fe-9625-b18de0d01b4e None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:31.613243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52fb374-e283-48fe-9625-b18de0d01b4e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:31.613622 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52fb374-e283-48fe-9625-b18de0d01b4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:31.632919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-a52fb374-e283-48fe-9625-b18de0d01b4e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:31.634107 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a52fb374-e283-48fe-9625-b18de0d01b4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:31.695780 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1320/2631] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:31.704032 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6364ddbe-f2a8-4b58-8612-6d402dcb5021 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:31.704321 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6364ddbe-f2a8-4b58-8612-6d402dcb5021 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:31.704597 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6364ddbe-f2a8-4b58-8612-6d402dcb5021 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:31.729344 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-6364ddbe-f2a8-4b58-8612-6d402dcb5021 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:31.730591 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-6364ddbe-f2a8-4b58-8612-6d402dcb5021 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:31.738545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1312/2632] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:31.747887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52fb374-e283-48fe-9625-b18de0d01b4e None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:31.757687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52fb374-e283-48fe-9625-b18de0d01b4e None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:31.762318 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24856863-17d1-43ff-a961-87dd57143638 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:31.762944 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24856863-17d1-43ff-a961-87dd57143638 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:31.763227 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24856863-17d1-43ff-a961-87dd57143638 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:31.763501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24856863-17d1-43ff-a961-87dd57143638 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:31.764151 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24856863-17d1-43ff-a961-87dd57143638 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:09:31.764956 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24856863-17d1-43ff-a961-87dd57143638 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:31.766586 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24856863-17d1-43ff-a961-87dd57143638 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:31.778477 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1321/2633] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:09:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:09:31.783435 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6364ddbe-f2a8-4b58-8612-6d402dcb5021 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:31.794360 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6364ddbe-f2a8-4b58-8612-6d402dcb5021 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:31.802696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20761d40-24b9-4052-ae72-cb206d921e12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:31.803309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20761d40-24b9-4052-ae72-cb206d921e12 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:31.803648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20761d40-24b9-4052-ae72-cb206d921e12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:31.803855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20761d40-24b9-4052-ae72-cb206d921e12 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:31.804486 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20761d40-24b9-4052-ae72-cb206d921e12 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:31.806511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20761d40-24b9-4052-ae72-cb206d921e12 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:31.954791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1313/2634] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:09:31 2022] POST /identity/v3/projects => generated 385 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:09:31.958927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24856863-17d1-43ff-a961-87dd57143638 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:31.968782 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24856863-17d1-43ff-a961-87dd57143638 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:32.027603 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6367ade-f931-4c28-9bbf-0429ae1b57f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:32.028204 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6367ade-f931-4c28-9bbf-0429ae1b57f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:32.028484 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6367ade-f931-4c28-9bbf-0429ae1b57f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:32.028752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6367ade-f931-4c28-9bbf-0429ae1b57f9 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:32.029376 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6367ade-f931-4c28-9bbf-0429ae1b57f9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:32.031896 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6367ade-f931-4c28-9bbf-0429ae1b57f9 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:32.233881 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1322/2635] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:09:31 2022] POST /identity/v3/users => generated 293 bytes in 275 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:09:32.238071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20761d40-24b9-4052-ae72-cb206d921e12 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:32.249581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20761d40-24b9-4052-ae72-cb206d921e12 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:32.254100 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544f47f9-1e7b-49a3-9a25-41a217e08b0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:32.254799 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544f47f9-1e7b-49a3-9a25-41a217e08b0a admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:32.255078 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544f47f9-1e7b-49a3-9a25-41a217e08b0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:32.255349 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544f47f9-1e7b-49a3-9a25-41a217e08b0a admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:32.256085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544f47f9-1e7b-49a3-9a25-41a217e08b0a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:32.257697 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544f47f9-1e7b-49a3-9a25-41a217e08b0a admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:32.271525 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1314/2636] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:09:32.272708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6367ade-f931-4c28-9bbf-0429ae1b57f9 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:32.282938 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6367ade-f931-4c28-9bbf-0429ae1b57f9 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:32.290598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2500b672-85b2-40bc-b47f-5cecefb07338 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:32.290598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2500b672-85b2-40bc-b47f-5cecefb07338 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:32.290598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2500b672-85b2-40bc-b47f-5cecefb07338 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:32.290598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2500b672-85b2-40bc-b47f-5cecefb07338 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:32.291072 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2500b672-85b2-40bc-b47f-5cecefb07338 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:32.291196 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2500b672-85b2-40bc-b47f-5cecefb07338 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:32.302962 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1323/2637] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:09:32.304307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:32.310884 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544f47f9-1e7b-49a3-9a25-41a217e08b0a admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:32.323316 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544f47f9-1e7b-49a3-9a25-41a217e08b0a admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:32.328555 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3bce25-46b6-4904-892f-cc800831b13a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:32.330913 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3bce25-46b6-4904-892f-cc800831b13a admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:32.330913 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3bce25-46b6-4904-892f-cc800831b13a admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:32.330913 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3bce25-46b6-4904-892f-cc800831b13a admin admin] PATH_INFO: `/v3/projects/c8e994ed4b7a47cfbef97e80d2f1c76d/users/affee49ab11a4ee9b89d30582779b509/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:32.346515 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3bce25-46b6-4904-892f-cc800831b13a admin admin] RBAC: Authorizing `identity:create_grant(project_id=c8e994ed4b7a47cfbef97e80d2f1c76d, user_id=affee49ab11a4ee9b89d30582779b509, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:32.349067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3bce25-46b6-4904-892f-cc800831b13a admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:32.426490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1315/2638] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:09:32 2022] PUT /identity/v3/projects/c8e994ed4b7a47cfbef97e80d2f1c76d/users/affee49ab11a4ee9b89d30582779b509/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:32.438208 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fc0bc3-1fb0-4511-bd2c-8800ffd580c3 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:32.438521 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fc0bc3-1fb0-4511-bd2c-8800ffd580c3 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:32.438786 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fc0bc3-1fb0-4511-bd2c-8800ffd580c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:32.469168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-97fc0bc3-1fb0-4511-bd2c-8800ffd580c3 None None] MFA Rules not processed for user `affee49ab11a4ee9b89d30582779b509`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:32.470435 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-97fc0bc3-1fb0-4511-bd2c-8800ffd580c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:32.533020 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1324/2639] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:32 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:32.537540 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3bce25-46b6-4904-892f-cc800831b13a admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:32.555046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:32.598665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3bce25-46b6-4904-892f-cc800831b13a admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:32.603488 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b14050b-cdf5-4923-b71d-9e136975ccb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:32.604083 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b14050b-cdf5-4923-b71d-9e136975ccb5 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:32.604362 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b14050b-cdf5-4923-b71d-9e136975ccb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:32.604632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b14050b-cdf5-4923-b71d-9e136975ccb5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:32.605251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b14050b-cdf5-4923-b71d-9e136975ccb5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:32.607167 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b14050b-cdf5-4923-b71d-9e136975ccb5 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:32.702755 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1316/2640] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:09:32 2022] POST /identity/v3/projects => generated 383 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:09:32.706959 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fc0bc3-1fb0-4511-bd2c-8800ffd580c3 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:32.717558 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fc0bc3-1fb0-4511-bd2c-8800ffd580c3 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:32.733897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:32.773219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5e9132-81e9-49d6-9097-8f8442fb0f66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:32.773816 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5e9132-81e9-49d6-9097-8f8442fb0f66 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:32.774096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5e9132-81e9-49d6-9097-8f8442fb0f66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:32.774395 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5e9132-81e9-49d6-9097-8f8442fb0f66 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:32.775019 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5e9132-81e9-49d6-9097-8f8442fb0f66 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:32.776953 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5e9132-81e9-49d6-9097-8f8442fb0f66 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:32.912706 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1325/2641] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:09:32 2022] POST /identity/v3/users => generated 292 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:09:32.916719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b14050b-cdf5-4923-b71d-9e136975ccb5 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:32.927315 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b14050b-cdf5-4923-b71d-9e136975ccb5 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:32.932166 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dcf3ac-d357-4207-8d10-cae9324ec8d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:32.933161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dcf3ac-d357-4207-8d10-cae9324ec8d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:32.933161 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dcf3ac-d357-4207-8d10-cae9324ec8d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:32.933395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85dcf3ac-d357-4207-8d10-cae9324ec8d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:32.933909 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85dcf3ac-d357-4207-8d10-cae9324ec8d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:32.935544 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85dcf3ac-d357-4207-8d10-cae9324ec8d7 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:32.947436 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1317/2642] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:09:32.955620 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5e9132-81e9-49d6-9097-8f8442fb0f66 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:32.966029 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5e9132-81e9-49d6-9097-8f8442fb0f66 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:32.971197 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263bdbe0-868a-4dcf-b9e5-78f3c033838e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:32.971796 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263bdbe0-868a-4dcf-b9e5-78f3c033838e admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:32.972075 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263bdbe0-868a-4dcf-b9e5-78f3c033838e admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:32.972345 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263bdbe0-868a-4dcf-b9e5-78f3c033838e admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:32.972988 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263bdbe0-868a-4dcf-b9e5-78f3c033838e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:32.974601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263bdbe0-868a-4dcf-b9e5-78f3c033838e admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:32.987309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1326/2643] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:32 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:09:32.998584 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dcf3ac-d357-4207-8d10-cae9324ec8d7 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:33.009068 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85dcf3ac-d357-4207-8d10-cae9324ec8d7 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:33.013761 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3a6949-2ccc-409e-8662-26c8090f1d2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:33.015066 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3a6949-2ccc-409e-8662-26c8090f1d2c admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:33.015352 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3a6949-2ccc-409e-8662-26c8090f1d2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:33.015752 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3a6949-2ccc-409e-8662-26c8090f1d2c admin admin] PATH_INFO: `/v3/projects/a94c46a0025243fbbc7b09035d2cb8b0/users/3e20dfc6e5704cbab362fdaee9dc9f0d/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:33.032016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3a6949-2ccc-409e-8662-26c8090f1d2c admin admin] RBAC: Authorizing `identity:create_grant(project_id=a94c46a0025243fbbc7b09035d2cb8b0, user_id=3e20dfc6e5704cbab362fdaee9dc9f0d, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:33.034677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3a6949-2ccc-409e-8662-26c8090f1d2c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:33.141593 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1318/2644] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:09:32 2022] PUT /identity/v3/projects/a94c46a0025243fbbc7b09035d2cb8b0/users/3e20dfc6e5704cbab362fdaee9dc9f0d/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:09:33.151958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b745b8-6e76-4b22-be32-c62662394dc1 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:33.152255 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b745b8-6e76-4b22-be32-c62662394dc1 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:33.152530 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b745b8-6e76-4b22-be32-c62662394dc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:33.181498 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-a2b745b8-6e76-4b22-be32-c62662394dc1 None None] MFA Rules not processed for user `3e20dfc6e5704cbab362fdaee9dc9f0d`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:33.182997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a2b745b8-6e76-4b22-be32-c62662394dc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:33.247653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1327/2645] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:33 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:33.253171 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3a6949-2ccc-409e-8662-26c8090f1d2c admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:33.313597 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3a6949-2ccc-409e-8662-26c8090f1d2c admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:33.318260 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69090793-ea6f-4ee9-a7ca-50d308c302ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:33.318883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69090793-ea6f-4ee9-a7ca-50d308c302ff admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:33.319163 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69090793-ea6f-4ee9-a7ca-50d308c302ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:33.319428 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69090793-ea6f-4ee9-a7ca-50d308c302ff admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:33.320055 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69090793-ea6f-4ee9-a7ca-50d308c302ff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:33.321925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69090793-ea6f-4ee9-a7ca-50d308c302ff admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:33.412645 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1319/2646] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:09:33 2022] POST /identity/v3/projects => generated 385 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:09:33.418367 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b745b8-6e76-4b22-be32-c62662394dc1 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:33.430153 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b745b8-6e76-4b22-be32-c62662394dc1 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:33.488350 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae15edc-f4d2-4f2d-a7f5-fa21725c851b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:33.488860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae15edc-f4d2-4f2d-a7f5-fa21725c851b admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:33.489141 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae15edc-f4d2-4f2d-a7f5-fa21725c851b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:33.489407 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae15edc-f4d2-4f2d-a7f5-fa21725c851b admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:33.490030 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae15edc-f4d2-4f2d-a7f5-fa21725c851b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:33.491964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae15edc-f4d2-4f2d-a7f5-fa21725c851b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:33.569341 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1328/2647] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:09:33 2022] POST /identity/v3/users => generated 299 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:09:33.573333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69090793-ea6f-4ee9-a7ca-50d308c302ff admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:33.583590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69090793-ea6f-4ee9-a7ca-50d308c302ff admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:33.588291 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5fa3ad-00fd-4bfa-a988-f2b0bae2b171 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:33.588886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5fa3ad-00fd-4bfa-a988-f2b0bae2b171 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:33.589162 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5fa3ad-00fd-4bfa-a988-f2b0bae2b171 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:33.589432 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5fa3ad-00fd-4bfa-a988-f2b0bae2b171 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:33.590068 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5fa3ad-00fd-4bfa-a988-f2b0bae2b171 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:33.591704 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5fa3ad-00fd-4bfa-a988-f2b0bae2b171 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:33.602926 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1320/2648] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:33 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:09:33.607327 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae15edc-f4d2-4f2d-a7f5-fa21725c851b admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:33.618474 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae15edc-f4d2-4f2d-a7f5-fa21725c851b admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:33.623128 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a593c6c-d3a3-438a-b4b5-fc020d84a00c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:33.623794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a593c6c-d3a3-438a-b4b5-fc020d84a00c admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:33.624071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a593c6c-d3a3-438a-b4b5-fc020d84a00c admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:33.624344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a593c6c-d3a3-438a-b4b5-fc020d84a00c admin admin] PATH_INFO: `/v3/projects/cc406d76de034732a1e8320781d0742a/users/cb0f225034f94be99b3a3c3c89fd5cb3/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:33.638761 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a593c6c-d3a3-438a-b4b5-fc020d84a00c admin admin] RBAC: Authorizing `identity:create_grant(project_id=cc406d76de034732a1e8320781d0742a, user_id=cb0f225034f94be99b3a3c3c89fd5cb3, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:33.641392 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a593c6c-d3a3-438a-b4b5-fc020d84a00c admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:33.643549 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:33.701254 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1329/2649] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:09:33 2022] PUT /identity/v3/projects/cc406d76de034732a1e8320781d0742a/users/cb0f225034f94be99b3a3c3c89fd5cb3/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:33.705313 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5fa3ad-00fd-4bfa-a988-f2b0bae2b171 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:33.715083 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5fa3ad-00fd-4bfa-a988-f2b0bae2b171 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:33.755322 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:33.775175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa58faf-7c61-437d-a908-a7db6b857886 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:33.775175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa58faf-7c61-437d-a908-a7db6b857886 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:33.775175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa58faf-7c61-437d-a908-a7db6b857886 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:33.775175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa58faf-7c61-437d-a908-a7db6b857886 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:33.775587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa58faf-7c61-437d-a908-a7db6b857886 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:33.775686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa58faf-7c61-437d-a908-a7db6b857886 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:33.785759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1321/2650] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:33 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:09:33.789869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a593c6c-d3a3-438a-b4b5-fc020d84a00c admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:33.802159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a593c6c-d3a3-438a-b4b5-fc020d84a00c admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:33.808506 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5ec89f-bbd5-4a0e-8647-3bf5d249f292 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:33.809319 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5ec89f-bbd5-4a0e-8647-3bf5d249f292 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:33.809751 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5ec89f-bbd5-4a0e-8647-3bf5d249f292 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:33.810071 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5ec89f-bbd5-4a0e-8647-3bf5d249f292 admin admin] PATH_INFO: `/v3/projects/cc406d76de034732a1e8320781d0742a/users/cb0f225034f94be99b3a3c3c89fd5cb3/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:33.813084 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5ec89f-bbd5-4a0e-8647-3bf5d249f292 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cc406d76de034732a1e8320781d0742a, user_id=cb0f225034f94be99b3a3c3c89fd5cb3, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:33.816064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5ec89f-bbd5-4a0e-8647-3bf5d249f292 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:33.828513 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1330/2651] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:09:33 2022] PUT /identity/v3/projects/cc406d76de034732a1e8320781d0742a/users/cb0f225034f94be99b3a3c3c89fd5cb3/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:33.838731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8c2143-457d-43c6-bf04-97be1105e88a None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:33.839019 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8c2143-457d-43c6-bf04-97be1105e88a None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:33.839300 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd8c2143-457d-43c6-bf04-97be1105e88a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:33.867159 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-fd8c2143-457d-43c6-bf04-97be1105e88a None None] MFA Rules not processed for user `cb0f225034f94be99b3a3c3c89fd5cb3`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:33.868390 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-fd8c2143-457d-43c6-bf04-97be1105e88a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:33.942449 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1322/2652] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:33 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:33.955320 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5ec89f-bbd5-4a0e-8647-3bf5d249f292 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:33.957144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-3f5ec89f-bbd5-4a0e-8647-3bf5d249f292 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:34.019467 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5ec89f-bbd5-4a0e-8647-3bf5d249f292 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:34.023163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6425f3d2-97b4-458a-86f2-aac55ea1a8ae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:34.023763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6425f3d2-97b4-458a-86f2-aac55ea1a8ae service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:34.024033 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6425f3d2-97b4-458a-86f2-aac55ea1a8ae service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:34.024301 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6425f3d2-97b4-458a-86f2-aac55ea1a8ae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:34.081005 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6425f3d2-97b4-458a-86f2-aac55ea1a8ae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:34.082600 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6425f3d2-97b4-458a-86f2-aac55ea1a8ae service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:34.091530 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1331/2653] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:33 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:38.094918 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8c2143-457d-43c6-bf04-97be1105e88a None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:38.104978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd8c2143-457d-43c6-bf04-97be1105e88a None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:38.109495 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91782ec3-2852-4626-b56a-69a49675fb92 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:38.110171 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91782ec3-2852-4626-b56a-69a49675fb92 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:38.110490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91782ec3-2852-4626-b56a-69a49675fb92 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:38.110769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91782ec3-2852-4626-b56a-69a49675fb92 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:38.180751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91782ec3-2852-4626-b56a-69a49675fb92 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:38.182250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91782ec3-2852-4626-b56a-69a49675fb92 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:38.192251 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1323/2654] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:38 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:39.726439 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6425f3d2-97b4-458a-86f2-aac55ea1a8ae service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:39.735694 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6425f3d2-97b4-458a-86f2-aac55ea1a8ae service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:39.739439 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc79c4b-9fae-41ad-afd5-6541445dec90 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:39.740029 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc79c4b-9fae-41ad-afd5-6541445dec90 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:39.740306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc79c4b-9fae-41ad-afd5-6541445dec90 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:39.740572 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc79c4b-9fae-41ad-afd5-6541445dec90 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:39.749561 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc79c4b-9fae-41ad-afd5-6541445dec90 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:39.751085 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc79c4b-9fae-41ad-afd5-6541445dec90 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:39.760430 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1332/2655] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:39 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:41.680332 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91782ec3-2852-4626-b56a-69a49675fb92 service cinder] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:41.681975 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-91782ec3-2852-4626-b56a-69a49675fb92 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:41.746819 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91782ec3-2852-4626-b56a-69a49675fb92 service cinder] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:41.752613 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a608e92f-dd8a-4e52-bff7-473f23d5868f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:41.753243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a608e92f-dd8a-4e52-bff7-473f23d5868f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:41.753523 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a608e92f-dd8a-4e52-bff7-473f23d5868f admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:41.753793 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a608e92f-dd8a-4e52-bff7-473f23d5868f admin admin] PATH_INFO: `/v3/users/c422c0fc22e9411e92f1945583082308` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:41.754953 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a608e92f-dd8a-4e52-bff7-473f23d5868f admin admin] RBAC: Authorizing `identity:delete_user(user_id=c422c0fc22e9411e92f1945583082308)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:41.756884 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a608e92f-dd8a-4e52-bff7-473f23d5868f admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:41.758789 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:41.991527 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a608e92f-dd8a-4e52-bff7-473f23d5868f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c422c0fc22e9411e92f1945583082308'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:42.061460 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a608e92f-dd8a-4e52-bff7-473f23d5868f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c422c0fc22e9411e92f1945583082308'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:42.062691 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a608e92f-dd8a-4e52-bff7-473f23d5868f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c422c0fc22e9411e92f1945583082308'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:42.075356 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a608e92f-dd8a-4e52-bff7-473f23d5868f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c422c0fc22e9411e92f1945583082308'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:42.083154 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a608e92f-dd8a-4e52-bff7-473f23d5868f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c422c0fc22e9411e92f1945583082308'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:42.090185 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1324/2656] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:09:41 2022] DELETE /identity/v3/users/c422c0fc22e9411e92f1945583082308 => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:42.094874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22056b34-2aa1-40ae-ae54-c289729b5521 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:42.095361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22056b34-2aa1-40ae-ae54-c289729b5521 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:42.095457 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22056b34-2aa1-40ae-ae54-c289729b5521 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:42.115891 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-22056b34-2aa1-40ae-ae54-c289729b5521 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:42.117141 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-22056b34-2aa1-40ae-ae54-c289729b5521 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:42.175117 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1333/2657] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:42.185234 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a608e92f-dd8a-4e52-bff7-473f23d5868f admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:42.186752 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a608e92f-dd8a-4e52-bff7-473f23d5868f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:42.247069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a608e92f-dd8a-4e52-bff7-473f23d5868f admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:42.251318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ab202f-9aa4-4e07-8b21-54a22187fd49 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:42.251917 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ab202f-9aa4-4e07-8b21-54a22187fd49 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:42.252194 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ab202f-9aa4-4e07-8b21-54a22187fd49 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:42.252464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ab202f-9aa4-4e07-8b21-54a22187fd49 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:42.261265 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ab202f-9aa4-4e07-8b21-54a22187fd49 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:42.262712 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ab202f-9aa4-4e07-8b21-54a22187fd49 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:42.272095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1325/2658] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:42.702771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22056b34-2aa1-40ae-ae54-c289729b5521 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:42.704902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-22056b34-2aa1-40ae-ae54-c289729b5521 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:42.706159 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ab202f-9aa4-4e07-8b21-54a22187fd49 service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:42.710515 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-54ab202f-9aa4-4e07-8b21-54a22187fd49 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:42.720301 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22056b34-2aa1-40ae-ae54-c289729b5521 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:42.724601 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499b1f82-b302-4ac8-9d03-a22df7d315a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:42.725222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499b1f82-b302-4ac8-9d03-a22df7d315a9 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:42.725501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499b1f82-b302-4ac8-9d03-a22df7d315a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:42.725773 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499b1f82-b302-4ac8-9d03-a22df7d315a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:42.727687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-499b1f82-b302-4ac8-9d03-a22df7d315a9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:42.733279 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ab202f-9aa4-4e07-8b21-54a22187fd49 service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:42.740736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bf2b40-2ebf-4574-88bb-d5531c589849 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:42.740736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bf2b40-2ebf-4574-88bb-d5531c589849 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:42.740736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bf2b40-2ebf-4574-88bb-d5531c589849 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:42.740736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bf2b40-2ebf-4574-88bb-d5531c589849 admin admin] PATH_INFO: `/v3/projects/da6f07b1dac849288b096121811d770a` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:42.742046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bf2b40-2ebf-4574-88bb-d5531c589849 admin admin] RBAC: Authorizing `identity:delete_project(project_id=da6f07b1dac849288b096121811d770a)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:42.744044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bf2b40-2ebf-4574-88bb-d5531c589849 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:42.752062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:42.795296 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499b1f82-b302-4ac8-9d03-a22df7d315a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:42.797676 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499b1f82-b302-4ac8-9d03-a22df7d315a9 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:42.808408 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1334/2659] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:42 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:42.947720 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e7bf2b40-2ebf-4574-88bb-d5531c589849 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'da6f07b1dac849288b096121811d770a', 'request_id': 'req-e7bf2b40-2ebf-4574-88bb-d5531c589849'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:43.088847 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e7bf2b40-2ebf-4574-88bb-d5531c589849 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'da6f07b1dac849288b096121811d770a', 'request_id': 'req-e7bf2b40-2ebf-4574-88bb-d5531c589849'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:43.097088 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e7bf2b40-2ebf-4574-88bb-d5531c589849 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'da6f07b1dac849288b096121811d770a', 'request_id': 'req-e7bf2b40-2ebf-4574-88bb-d5531c589849'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:43.110166 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e7bf2b40-2ebf-4574-88bb-d5531c589849 admin admin] The token cache is being invalidate because project da6f07b1dac849288b096121811d770a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:09:43.110511 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e7bf2b40-2ebf-4574-88bb-d5531c589849 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:43.111189 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e7bf2b40-2ebf-4574-88bb-d5531c589849 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:43.114025 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1326/2660] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:09:42 2022] DELETE /identity/v3/projects/da6f07b1dac849288b096121811d770a => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:43.123841 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a09d86f-c4b2-44bc-ac30-a6448b15156e None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:43.124312 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a09d86f-c4b2-44bc-ac30-a6448b15156e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:43.124416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a09d86f-c4b2-44bc-ac30-a6448b15156e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:43.145506 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-1a09d86f-c4b2-44bc-ac30-a6448b15156e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:43.146944 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-1a09d86f-c4b2-44bc-ac30-a6448b15156e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:43.218306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1335/2661] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:43.227105 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c06e692-319d-4df3-9458-df4f7f484596 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:43.227465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c06e692-319d-4df3-9458-df4f7f484596 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:43.227749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c06e692-319d-4df3-9458-df4f7f484596 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:43.237055 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:43.248377 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-2c06e692-319d-4df3-9458-df4f7f484596 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:43.251847 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2c06e692-319d-4df3-9458-df4f7f484596 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:43.258786 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1327/2662] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:43.263120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a09d86f-c4b2-44bc-ac30-a6448b15156e None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:43.273929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a09d86f-c4b2-44bc-ac30-a6448b15156e None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:43.283564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d38f8be-75bc-4699-b1ec-183e8a207416 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:43.283564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d38f8be-75bc-4699-b1ec-183e8a207416 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:43.283564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d38f8be-75bc-4699-b1ec-183e8a207416 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:43.283564 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d38f8be-75bc-4699-b1ec-183e8a207416 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:43.283914 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d38f8be-75bc-4699-b1ec-183e8a207416 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:09:43.283914 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d38f8be-75bc-4699-b1ec-183e8a207416 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:43.283914 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d38f8be-75bc-4699-b1ec-183e8a207416 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:43.294421 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1336/2663] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:09:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:09:43.299593 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c06e692-319d-4df3-9458-df4f7f484596 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:43.310608 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c06e692-319d-4df3-9458-df4f7f484596 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:43.315954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb20423-09d7-496a-b53c-958d3bebeb88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:43.316556 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb20423-09d7-496a-b53c-958d3bebeb88 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:43.316834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb20423-09d7-496a-b53c-958d3bebeb88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:43.317102 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb20423-09d7-496a-b53c-958d3bebeb88 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:43.317724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb20423-09d7-496a-b53c-958d3bebeb88 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:43.319765 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bb20423-09d7-496a-b53c-958d3bebeb88 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:43.399087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1328/2664] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:09:43 2022] POST /identity/v3/projects => generated 361 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:09:43.400167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d38f8be-75bc-4699-b1ec-183e8a207416 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:43.411056 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d38f8be-75bc-4699-b1ec-183e8a207416 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:43.484227 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8e21a6-1178-4160-969c-d1047e71ef58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:43.484227 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8e21a6-1178-4160-969c-d1047e71ef58 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:43.484227 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8e21a6-1178-4160-969c-d1047e71ef58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:43.484227 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8e21a6-1178-4160-969c-d1047e71ef58 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:43.484554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8e21a6-1178-4160-969c-d1047e71ef58 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:43.485605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8e21a6-1178-4160-969c-d1047e71ef58 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:43.559309 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1337/2665] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:09:43 2022] POST /identity/v3/users => generated 281 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:09:43.564058 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb20423-09d7-496a-b53c-958d3bebeb88 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:43.574485 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb20423-09d7-496a-b53c-958d3bebeb88 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:43.579327 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f364894-12c6-437e-986b-09e88f5ab592 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:43.580012 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f364894-12c6-437e-986b-09e88f5ab592 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:43.580286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f364894-12c6-437e-986b-09e88f5ab592 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:43.580554 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f364894-12c6-437e-986b-09e88f5ab592 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:43.581796 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f364894-12c6-437e-986b-09e88f5ab592 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:43.582921 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f364894-12c6-437e-986b-09e88f5ab592 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:43.594213 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1329/2666] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:43 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:09:43.598579 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8e21a6-1178-4160-969c-d1047e71ef58 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:43.610263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8e21a6-1178-4160-969c-d1047e71ef58 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:43.614925 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d726f23-ac30-44d9-b1d5-8343983e7272 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:43.615523 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d726f23-ac30-44d9-b1d5-8343983e7272 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:43.615881 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d726f23-ac30-44d9-b1d5-8343983e7272 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:43.616159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d726f23-ac30-44d9-b1d5-8343983e7272 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:43.616794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d726f23-ac30-44d9-b1d5-8343983e7272 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:43.618455 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d726f23-ac30-44d9-b1d5-8343983e7272 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:43.629762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1338/2667] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:43 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:09:43.634217 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f364894-12c6-437e-986b-09e88f5ab592 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:43.644533 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f364894-12c6-437e-986b-09e88f5ab592 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:43.650981 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da4eeb3-167c-42a3-88d9-06c94a9860ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:43.651651 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da4eeb3-167c-42a3-88d9-06c94a9860ce admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:43.651932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da4eeb3-167c-42a3-88d9-06c94a9860ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:43.652203 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da4eeb3-167c-42a3-88d9-06c94a9860ce admin admin] PATH_INFO: `/v3/projects/2614318f636e4c3d90d5594250e544dd/users/658ec5f160b943c88e9bfab969158a92/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:43.667658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da4eeb3-167c-42a3-88d9-06c94a9860ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=2614318f636e4c3d90d5594250e544dd, user_id=658ec5f160b943c88e9bfab969158a92, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:43.677141 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da4eeb3-167c-42a3-88d9-06c94a9860ce admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:43.724856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1330/2668] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:09:43 2022] PUT /identity/v3/projects/2614318f636e4c3d90d5594250e544dd/users/658ec5f160b943c88e9bfab969158a92/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:43.735698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e54c32-454a-433c-b435-9e76254c7039 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:43.735985 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e54c32-454a-433c-b435-9e76254c7039 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:43.736258 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e54c32-454a-433c-b435-9e76254c7039 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:43.771254 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-69e54c32-454a-433c-b435-9e76254c7039 None None] MFA Rules not processed for user `658ec5f160b943c88e9bfab969158a92`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:43.772510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-69e54c32-454a-433c-b435-9e76254c7039 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:43.782195 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:43.838814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1339/2669] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:43 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:43.880074 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da4eeb3-167c-42a3-88d9-06c94a9860ce admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:43.881781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2da4eeb3-167c-42a3-88d9-06c94a9860ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:43.942275 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da4eeb3-167c-42a3-88d9-06c94a9860ce admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:43.946026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3ef04f-640f-4df7-a557-4c4fbd922ff2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:43.946675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3ef04f-640f-4df7-a557-4c4fbd922ff2 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:43.947064 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3ef04f-640f-4df7-a557-4c4fbd922ff2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:43.947336 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3ef04f-640f-4df7-a557-4c4fbd922ff2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:43.956133 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3ef04f-640f-4df7-a557-4c4fbd922ff2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:43.957885 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3ef04f-640f-4df7-a557-4c4fbd922ff2 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:43.966658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1331/2670] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:43 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:43.983242 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e54c32-454a-433c-b435-9e76254c7039 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:44.049893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e54c32-454a-433c-b435-9e76254c7039 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:44.054526 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88d3898-7b2c-4bb0-ac76-5c08afa142ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:44.055149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88d3898-7b2c-4bb0-ac76-5c08afa142ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:44.055439 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88d3898-7b2c-4bb0-ac76-5c08afa142ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:44.055698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88d3898-7b2c-4bb0-ac76-5c08afa142ad admin admin] PATH_INFO: `/v3/users/658ec5f160b943c88e9bfab969158a92` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:44.057090 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88d3898-7b2c-4bb0-ac76-5c08afa142ad admin admin] RBAC: Authorizing `identity:delete_user(user_id=658ec5f160b943c88e9bfab969158a92)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:44.059066 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88d3898-7b2c-4bb0-ac76-5c08afa142ad admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:44.265394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-c88d3898-7b2c-4bb0-ac76-5c08afa142ad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '658ec5f160b943c88e9bfab969158a92'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:44.347407 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-c88d3898-7b2c-4bb0-ac76-5c08afa142ad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '658ec5f160b943c88e9bfab969158a92'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:44.361358 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-c88d3898-7b2c-4bb0-ac76-5c08afa142ad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '658ec5f160b943c88e9bfab969158a92'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:44.362040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-c88d3898-7b2c-4bb0-ac76-5c08afa142ad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '658ec5f160b943c88e9bfab969158a92'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:44.369375 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-c88d3898-7b2c-4bb0-ac76-5c08afa142ad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '658ec5f160b943c88e9bfab969158a92'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:44.377034 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1340/2671] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:09:43 2022] DELETE /identity/v3/users/658ec5f160b943c88e9bfab969158a92 => generated 0 bytes in 394 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:44.381454 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ab351d-1aa4-4fd2-9e0a-52373b0be55b None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:44.381737 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ab351d-1aa4-4fd2-9e0a-52373b0be55b None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:44.382011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ab351d-1aa4-4fd2-9e0a-52373b0be55b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:44.401877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-89ab351d-1aa4-4fd2-9e0a-52373b0be55b None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:44.403144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-89ab351d-1aa4-4fd2-9e0a-52373b0be55b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:44.470494 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1332/2672] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:09:44.484574 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88d3898-7b2c-4bb0-ac76-5c08afa142ad admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:44.486296 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c88d3898-7b2c-4bb0-ac76-5c08afa142ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:44.528547 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:44.535992 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ab351d-1aa4-4fd2-9e0a-52373b0be55b None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:44.537322 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-89ab351d-1aa4-4fd2-9e0a-52373b0be55b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:44.550498 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ab351d-1aa4-4fd2-9e0a-52373b0be55b None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:44.554443 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88d3898-7b2c-4bb0-ac76-5c08afa142ad admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:44.555302 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f885fed3-84fb-4b5c-8af0-3b93eeff617c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:44.555989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f885fed3-84fb-4b5c-8af0-3b93eeff617c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:44.556313 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f885fed3-84fb-4b5c-8af0-3b93eeff617c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:44.556528 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f885fed3-84fb-4b5c-8af0-3b93eeff617c admin admin] PATH_INFO: `/v3/users/affee49ab11a4ee9b89d30582779b509` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:44.558031 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f885fed3-84fb-4b5c-8af0-3b93eeff617c admin admin] RBAC: Authorizing `identity:delete_user(user_id=affee49ab11a4ee9b89d30582779b509)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:44.560049 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea735497-45a6-41cd-87d8-23138c47ed6f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:44.560199 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f885fed3-84fb-4b5c-8af0-3b93eeff617c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:44.560841 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea735497-45a6-41cd-87d8-23138c47ed6f service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:44.561222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea735497-45a6-41cd-87d8-23138c47ed6f service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:44.561595 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea735497-45a6-41cd-87d8-23138c47ed6f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:44.571998 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea735497-45a6-41cd-87d8-23138c47ed6f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:44.573435 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea735497-45a6-41cd-87d8-23138c47ed6f service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:44.586628 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1341/2673] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:44.722119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=93,keystone:INSERT=2,keystone:DELETE=5 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:44.727749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f885fed3-84fb-4b5c-8af0-3b93eeff617c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'affee49ab11a4ee9b89d30582779b509'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:44.810371 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f885fed3-84fb-4b5c-8af0-3b93eeff617c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'affee49ab11a4ee9b89d30582779b509'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:44.811228 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f885fed3-84fb-4b5c-8af0-3b93eeff617c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'affee49ab11a4ee9b89d30582779b509'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:44.824982 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f885fed3-84fb-4b5c-8af0-3b93eeff617c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'affee49ab11a4ee9b89d30582779b509'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:44.832934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-f885fed3-84fb-4b5c-8af0-3b93eeff617c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'affee49ab11a4ee9b89d30582779b509'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:44.839576 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1333/2674] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:09:44 2022] DELETE /identity/v3/users/affee49ab11a4ee9b89d30582779b509 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:09:44.844371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd9876f-5cad-4efa-ae45-0477d4ba3bb9 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:44.844729 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd9876f-5cad-4efa-ae45-0477d4ba3bb9 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:44.845074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd9876f-5cad-4efa-ae45-0477d4ba3bb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:44.866007 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-edd9876f-5cad-4efa-ae45-0477d4ba3bb9 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:44.867424 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-edd9876f-5cad-4efa-ae45-0477d4ba3bb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:44.962096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1342/2675] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:44.970700 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f885fed3-84fb-4b5c-8af0-3b93eeff617c admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:44.974271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f885fed3-84fb-4b5c-8af0-3b93eeff617c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:45.061453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f885fed3-84fb-4b5c-8af0-3b93eeff617c admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:45.065296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7152ba20-3c0b-403f-9335-8d93b3bfb59c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:45.065911 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7152ba20-3c0b-403f-9335-8d93b3bfb59c service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:45.066186 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7152ba20-3c0b-403f-9335-8d93b3bfb59c service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:45.066491 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7152ba20-3c0b-403f-9335-8d93b3bfb59c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:45.077608 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7152ba20-3c0b-403f-9335-8d93b3bfb59c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:45.079431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7152ba20-3c0b-403f-9335-8d93b3bfb59c service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:45.089879 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1334/2676] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:45.264709 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd9876f-5cad-4efa-ae45-0477d4ba3bb9 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:45.266207 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-edd9876f-5cad-4efa-ae45-0477d4ba3bb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:45.278847 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd9876f-5cad-4efa-ae45-0477d4ba3bb9 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:45.284218 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c560ded-258d-47ec-a1fa-817b05d154a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:45.284928 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c560ded-258d-47ec-a1fa-817b05d154a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:45.285194 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c560ded-258d-47ec-a1fa-817b05d154a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:45.285452 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c560ded-258d-47ec-a1fa-817b05d154a3 admin admin] PATH_INFO: `/v3/projects/2614318f636e4c3d90d5594250e544dd` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:45.286644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c560ded-258d-47ec-a1fa-817b05d154a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2614318f636e4c3d90d5594250e544dd)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:45.288621 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c560ded-258d-47ec-a1fa-817b05d154a3 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:45.412306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-1c560ded-258d-47ec-a1fa-817b05d154a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2614318f636e4c3d90d5594250e544dd', 'request_id': 'req-1c560ded-258d-47ec-a1fa-817b05d154a3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:45.507150 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-1c560ded-258d-47ec-a1fa-817b05d154a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2614318f636e4c3d90d5594250e544dd', 'request_id': 'req-1c560ded-258d-47ec-a1fa-817b05d154a3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:45.515321 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-1c560ded-258d-47ec-a1fa-817b05d154a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2614318f636e4c3d90d5594250e544dd', 'request_id': 'req-1c560ded-258d-47ec-a1fa-817b05d154a3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:45.527138 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-1c560ded-258d-47ec-a1fa-817b05d154a3 admin admin] The token cache is being invalidate because project 2614318f636e4c3d90d5594250e544dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:09:45.527511 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-1c560ded-258d-47ec-a1fa-817b05d154a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:45.528374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-1c560ded-258d-47ec-a1fa-817b05d154a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:45.530169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1343/2677] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:09:45 2022] DELETE /identity/v3/projects/2614318f636e4c3d90d5594250e544dd => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:45.535244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525d93ac-c583-41a3-91b5-9d44bc1946ae None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:45.535758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525d93ac-c583-41a3-91b5-9d44bc1946ae None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:45.536135 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525d93ac-c583-41a3-91b5-9d44bc1946ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:45.560191 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-525d93ac-c583-41a3-91b5-9d44bc1946ae None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:45.560191 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-525d93ac-c583-41a3-91b5-9d44bc1946ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:45.640227 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1335/2678] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:09:45.648118 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea90c9e-9654-44ae-a7fd-9d9211ccf6f0 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:45.648472 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea90c9e-9654-44ae-a7fd-9d9211ccf6f0 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:45.648814 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea90c9e-9654-44ae-a7fd-9d9211ccf6f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:45.674711 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-4ea90c9e-9654-44ae-a7fd-9d9211ccf6f0 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:45.676233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-4ea90c9e-9654-44ae-a7fd-9d9211ccf6f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:45.689581 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1344/2679] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:45.690792 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525d93ac-c583-41a3-91b5-9d44bc1946ae None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:45.701660 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525d93ac-c583-41a3-91b5-9d44bc1946ae None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:45.710798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9178584b-4d1e-4ec5-a37d-9824f3c7231c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:45.711480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9178584b-4d1e-4ec5-a37d-9824f3c7231c admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:45.711749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9178584b-4d1e-4ec5-a37d-9824f3c7231c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:45.712032 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9178584b-4d1e-4ec5-a37d-9824f3c7231c admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:45.712739 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9178584b-4d1e-4ec5-a37d-9824f3c7231c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:09:45.713036 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9178584b-4d1e-4ec5-a37d-9824f3c7231c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:45.714824 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9178584b-4d1e-4ec5-a37d-9824f3c7231c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:45.729164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1336/2680] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:09:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 23 14:09:45.734799 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea90c9e-9654-44ae-a7fd-9d9211ccf6f0 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:45.744127 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea90c9e-9654-44ae-a7fd-9d9211ccf6f0 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:45.749166 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f63e7aa-ae29-4959-98e3-b779dc1febd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:45.749818 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f63e7aa-ae29-4959-98e3-b779dc1febd9 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:45.750282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f63e7aa-ae29-4959-98e3-b779dc1febd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:45.750608 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f63e7aa-ae29-4959-98e3-b779dc1febd9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:45.751296 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f63e7aa-ae29-4959-98e3-b779dc1febd9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:45.753318 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f63e7aa-ae29-4959-98e3-b779dc1febd9 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:45.801957 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9178584b-4d1e-4ec5-a37d-9824f3c7231c admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:45.803989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9178584b-4d1e-4ec5-a37d-9824f3c7231c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:45.820660 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9178584b-4d1e-4ec5-a37d-9824f3c7231c admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:45.827132 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6447bc-2be4-4651-9a2c-e935a10935db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:45.827620 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6447bc-2be4-4651-9a2c-e935a10935db admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:45.828466 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6447bc-2be4-4651-9a2c-e935a10935db admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:45.828740 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6447bc-2be4-4651-9a2c-e935a10935db admin admin] PATH_INFO: `/v3/projects/c8e994ed4b7a47cfbef97e80d2f1c76d` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:45.829986 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6447bc-2be4-4651-9a2c-e935a10935db admin admin] RBAC: Authorizing `identity:delete_project(project_id=c8e994ed4b7a47cfbef97e80d2f1c76d)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:45.831943 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6447bc-2be4-4651-9a2c-e935a10935db admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:45.848801 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1345/2681] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:09:45 2022] POST /identity/v3/projects => generated 375 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:09:45.853238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f63e7aa-ae29-4959-98e3-b779dc1febd9 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:45.889852 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:45.925198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f63e7aa-ae29-4959-98e3-b779dc1febd9 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:45.930020 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680d7f6f-60ce-47b6-9bad-d6dffc322c18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:45.931289 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680d7f6f-60ce-47b6-9bad-d6dffc322c18 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:45.931574 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680d7f6f-60ce-47b6-9bad-d6dffc322c18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:45.932096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680d7f6f-60ce-47b6-9bad-d6dffc322c18 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:45.932728 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-680d7f6f-60ce-47b6-9bad-d6dffc322c18 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:45.935081 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-680d7f6f-60ce-47b6-9bad-d6dffc322c18 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:45.950564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:45.958067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-8a6447bc-2be4-4651-9a2c-e935a10935db admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c8e994ed4b7a47cfbef97e80d2f1c76d', 'request_id': 'req-8a6447bc-2be4-4651-9a2c-e935a10935db'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:46.050808 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1346/2682] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:09:45 2022] POST /identity/v3/users => generated 288 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:09:46.054681 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680d7f6f-60ce-47b6-9bad-d6dffc322c18 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:46.064916 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680d7f6f-60ce-47b6-9bad-d6dffc322c18 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:46.066982 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-8a6447bc-2be4-4651-9a2c-e935a10935db admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c8e994ed4b7a47cfbef97e80d2f1c76d', 'request_id': 'req-8a6447bc-2be4-4651-9a2c-e935a10935db'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:46.073172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-8a6447bc-2be4-4651-9a2c-e935a10935db admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c8e994ed4b7a47cfbef97e80d2f1c76d', 'request_id': 'req-8a6447bc-2be4-4651-9a2c-e935a10935db'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:46.085132 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-8a6447bc-2be4-4651-9a2c-e935a10935db admin admin] The token cache is being invalidate because project c8e994ed4b7a47cfbef97e80d2f1c76d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:09:46.085418 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-8a6447bc-2be4-4651-9a2c-e935a10935db admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:46.086202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-8a6447bc-2be4-4651-9a2c-e935a10935db admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:46.087751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1337/2683] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:09:45 2022] DELETE /identity/v3/projects/c8e994ed4b7a47cfbef97e80d2f1c76d => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:46.091676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6447bc-2be4-4651-9a2c-e935a10935db admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:46.093029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8a6447bc-2be4-4651-9a2c-e935a10935db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:46.124189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b29591-69c8-4672-abcb-7ac6ec035502 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:46.126212 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b29591-69c8-4672-abcb-7ac6ec035502 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:46.126535 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b29591-69c8-4672-abcb-7ac6ec035502 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:46.126958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b29591-69c8-4672-abcb-7ac6ec035502 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:46.127634 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b29591-69c8-4672-abcb-7ac6ec035502 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:46.129523 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b29591-69c8-4672-abcb-7ac6ec035502 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:46.143969 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1347/2684] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:46 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:09:46.145881 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b29591-69c8-4672-abcb-7ac6ec035502 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:46.148457 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-99b29591-69c8-4672-abcb-7ac6ec035502 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:46.166832 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6447bc-2be4-4651-9a2c-e935a10935db admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:46.168369 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b29591-69c8-4672-abcb-7ac6ec035502 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:46.173697 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11052798-aca6-4984-96e8-e1ac82ce737c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:46.174018 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9614bebc-7ec5-459a-9231-82fb701e5400 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:46.174423 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11052798-aca6-4984-96e8-e1ac82ce737c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:46.174655 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9614bebc-7ec5-459a-9231-82fb701e5400 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:46.174754 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11052798-aca6-4984-96e8-e1ac82ce737c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:46.175038 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9614bebc-7ec5-459a-9231-82fb701e5400 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:46.175144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11052798-aca6-4984-96e8-e1ac82ce737c admin admin] PATH_INFO: `/v3/users/3e20dfc6e5704cbab362fdaee9dc9f0d` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:46.175235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9614bebc-7ec5-459a-9231-82fb701e5400 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:46.175833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9614bebc-7ec5-459a-9231-82fb701e5400 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:46.176314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11052798-aca6-4984-96e8-e1ac82ce737c admin admin] RBAC: Authorizing `identity:delete_user(user_id=3e20dfc6e5704cbab362fdaee9dc9f0d)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:46.177415 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9614bebc-7ec5-459a-9231-82fb701e5400 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:46.178465 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11052798-aca6-4984-96e8-e1ac82ce737c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:46.188996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1348/2685] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:46 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:09:46.193065 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9614bebc-7ec5-459a-9231-82fb701e5400 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:46.202766 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9614bebc-7ec5-459a-9231-82fb701e5400 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:46.209604 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408c96ac-b3ba-4f22-bce1-8988cbb6dd4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:46.210567 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408c96ac-b3ba-4f22-bce1-8988cbb6dd4d admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:46.210842 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408c96ac-b3ba-4f22-bce1-8988cbb6dd4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:46.211542 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408c96ac-b3ba-4f22-bce1-8988cbb6dd4d admin admin] PATH_INFO: `/v3/projects/e56e32789db6408db7088a5c6b387267/users/edd988c9077142d6bdf6e93f16a1ad8e/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:46.226824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408c96ac-b3ba-4f22-bce1-8988cbb6dd4d admin admin] RBAC: Authorizing `identity:create_grant(project_id=e56e32789db6408db7088a5c6b387267, user_id=edd988c9077142d6bdf6e93f16a1ad8e, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:46.229522 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408c96ac-b3ba-4f22-bce1-8988cbb6dd4d admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:46.307420 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1349/2686] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:09:46 2022] PUT /identity/v3/projects/e56e32789db6408db7088a5c6b387267/users/edd988c9077142d6bdf6e93f16a1ad8e/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:09:46.318615 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebaf5df-1163-486b-95d4-76ca41ce9dc8 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:46.318900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebaf5df-1163-486b-95d4-76ca41ce9dc8 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:46.319198 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebaf5df-1163-486b-95d4-76ca41ce9dc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:46.347616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-cebaf5df-1163-486b-95d4-76ca41ce9dc8 None None] MFA Rules not processed for user `edd988c9077142d6bdf6e93f16a1ad8e`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:46.348786 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-11052798-aca6-4984-96e8-e1ac82ce737c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3e20dfc6e5704cbab362fdaee9dc9f0d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:46.348940 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-cebaf5df-1163-486b-95d4-76ca41ce9dc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:46.414180 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-11052798-aca6-4984-96e8-e1ac82ce737c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3e20dfc6e5704cbab362fdaee9dc9f0d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:46.414850 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-11052798-aca6-4984-96e8-e1ac82ce737c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3e20dfc6e5704cbab362fdaee9dc9f0d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:46.415498 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1350/2687] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:46 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:09:46.419991 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebaf5df-1163-486b-95d4-76ca41ce9dc8 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:46.429100 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-11052798-aca6-4984-96e8-e1ac82ce737c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3e20dfc6e5704cbab362fdaee9dc9f0d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:46.429536 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:46.437693 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-11052798-aca6-4984-96e8-e1ac82ce737c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3e20dfc6e5704cbab362fdaee9dc9f0d'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:46.444151 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1338/2688] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:09:46 2022] DELETE /identity/v3/users/3e20dfc6e5704cbab362fdaee9dc9f0d => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:09:46.487242 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebaf5df-1163-486b-95d4-76ca41ce9dc8 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:46.545300 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f4e662-9544-404a-87be-2d86196c5504 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:46.545892 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f4e662-9544-404a-87be-2d86196c5504 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:46.546166 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f4e662-9544-404a-87be-2d86196c5504 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:46.546967 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f4e662-9544-404a-87be-2d86196c5504 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:46.547732 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f4e662-9544-404a-87be-2d86196c5504 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:46.549600 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f4e662-9544-404a-87be-2d86196c5504 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:46.644921 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1351/2689] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:09:46 2022] POST /identity/v3/projects => generated 375 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:09:46.649020 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11052798-aca6-4984-96e8-e1ac82ce737c admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:46.650454 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-11052798-aca6-4984-96e8-e1ac82ce737c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:46.664361 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11052798-aca6-4984-96e8-e1ac82ce737c admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:46.711962 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:46.723966 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f694459-bdb0-48ba-a317-2bbb58062088 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:46.724580 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f694459-bdb0-48ba-a317-2bbb58062088 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:46.724852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f694459-bdb0-48ba-a317-2bbb58062088 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:46.725120 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f694459-bdb0-48ba-a317-2bbb58062088 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:46.725758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f694459-bdb0-48ba-a317-2bbb58062088 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:46.727691 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f694459-bdb0-48ba-a317-2bbb58062088 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:46.821224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1339/2690] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:09:46 2022] POST /identity/v3/users => generated 288 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:09:46.825476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f4e662-9544-404a-87be-2d86196c5504 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:46.835912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f4e662-9544-404a-87be-2d86196c5504 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:46.840654 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6bd2f2-9fd6-4e30-9ef8-6f6855db61b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:46.841244 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6bd2f2-9fd6-4e30-9ef8-6f6855db61b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:46.841540 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6bd2f2-9fd6-4e30-9ef8-6f6855db61b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:46.841794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6bd2f2-9fd6-4e30-9ef8-6f6855db61b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:46.842504 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6bd2f2-9fd6-4e30-9ef8-6f6855db61b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:46.844371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6bd2f2-9fd6-4e30-9ef8-6f6855db61b3 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:46.860504 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1352/2691] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:46 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:09:46.864289 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f694459-bdb0-48ba-a317-2bbb58062088 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:46.864886 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6bd2f2-9fd6-4e30-9ef8-6f6855db61b3 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:46.866209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8f694459-bdb0-48ba-a317-2bbb58062088 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:46.878263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6bd2f2-9fd6-4e30-9ef8-6f6855db61b3 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:46.882100 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f694459-bdb0-48ba-a317-2bbb58062088 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:46.883625 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6464b5-d0f8-4f1e-a667-8d4ff52179cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:46.884226 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6464b5-d0f8-4f1e-a667-8d4ff52179cc admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:46.884500 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6464b5-d0f8-4f1e-a667-8d4ff52179cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:46.884766 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6464b5-d0f8-4f1e-a667-8d4ff52179cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:46.885401 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6464b5-d0f8-4f1e-a667-8d4ff52179cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:46.886900 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e46a6d-a686-4220-bfa5-79f47ec2f258 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:46.887067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6464b5-d0f8-4f1e-a667-8d4ff52179cc admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:46.887531 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e46a6d-a686-4220-bfa5-79f47ec2f258 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:46.887839 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e46a6d-a686-4220-bfa5-79f47ec2f258 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:46.888131 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e46a6d-a686-4220-bfa5-79f47ec2f258 admin admin] PATH_INFO: `/v3/projects/a94c46a0025243fbbc7b09035d2cb8b0` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:46.889998 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e46a6d-a686-4220-bfa5-79f47ec2f258 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a94c46a0025243fbbc7b09035d2cb8b0)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:46.891973 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e46a6d-a686-4220-bfa5-79f47ec2f258 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:46.897673 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1353/2692] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:46 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:09:46.901833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6464b5-d0f8-4f1e-a667-8d4ff52179cc admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:46.912349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6464b5-d0f8-4f1e-a667-8d4ff52179cc admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:46.916955 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0248e345-9abb-4335-a969-92b9807dfc82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:46.917626 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0248e345-9abb-4335-a969-92b9807dfc82 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:46.917906 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0248e345-9abb-4335-a969-92b9807dfc82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:46.918175 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0248e345-9abb-4335-a969-92b9807dfc82 admin admin] PATH_INFO: `/v3/projects/d8e428a1de6f4533a4bc50d7a4a8894f/users/47a7beead42f450a9e1840c9e9563164/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:46.934596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0248e345-9abb-4335-a969-92b9807dfc82 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d8e428a1de6f4533a4bc50d7a4a8894f, user_id=47a7beead42f450a9e1840c9e9563164, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:46.937580 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0248e345-9abb-4335-a969-92b9807dfc82 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:47.005478 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1354/2693] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:09:46 2022] PUT /identity/v3/projects/d8e428a1de6f4533a4bc50d7a4a8894f/users/47a7beead42f450a9e1840c9e9563164/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:09:47.016367 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619d1e8f-3523-4da4-b8c6-f89a122f7012 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:47.016543 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-c0e46a6d-a686-4220-bfa5-79f47ec2f258 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a94c46a0025243fbbc7b09035d2cb8b0', 'request_id': 'req-c0e46a6d-a686-4220-bfa5-79f47ec2f258'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:47.016670 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619d1e8f-3523-4da4-b8c6-f89a122f7012 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:47.016912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619d1e8f-3523-4da4-b8c6-f89a122f7012 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:47.045661 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-619d1e8f-3523-4da4-b8c6-f89a122f7012 None None] MFA Rules not processed for user `47a7beead42f450a9e1840c9e9563164`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:47.047951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-619d1e8f-3523-4da4-b8c6-f89a122f7012 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:47.089936 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-c0e46a6d-a686-4220-bfa5-79f47ec2f258 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a94c46a0025243fbbc7b09035d2cb8b0', 'request_id': 'req-c0e46a6d-a686-4220-bfa5-79f47ec2f258'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:47.091730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:47.097202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-c0e46a6d-a686-4220-bfa5-79f47ec2f258 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a94c46a0025243fbbc7b09035d2cb8b0', 'request_id': 'req-c0e46a6d-a686-4220-bfa5-79f47ec2f258'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:47.109938 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-c0e46a6d-a686-4220-bfa5-79f47ec2f258 admin admin] The token cache is being invalidate because project a94c46a0025243fbbc7b09035d2cb8b0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:09:47.110361 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-c0e46a6d-a686-4220-bfa5-79f47ec2f258 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:47.110992 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-c0e46a6d-a686-4220-bfa5-79f47ec2f258 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:47.112506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1340/2694] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:09:46 2022] DELETE /identity/v3/projects/a94c46a0025243fbbc7b09035d2cb8b0 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:47.116351 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e46a6d-a686-4220-bfa5-79f47ec2f258 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:47.117644 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-c0e46a6d-a686-4220-bfa5-79f47ec2f258 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:47.118348 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1355/2695] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:47 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:09:47.122697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619d1e8f-3523-4da4-b8c6-f89a122f7012 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:47.184529 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e46a6d-a686-4220-bfa5-79f47ec2f258 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:47.185970 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619d1e8f-3523-4da4-b8c6-f89a122f7012 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:47.189269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e02e5ed-3cbf-4644-a657-5c4102a4a063 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:47.190044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e02e5ed-3cbf-4644-a657-5c4102a4a063 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:47.190415 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e02e5ed-3cbf-4644-a657-5c4102a4a063 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:47.190702 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e02e5ed-3cbf-4644-a657-5c4102a4a063 admin admin] PATH_INFO: `/v3/users/cb0f225034f94be99b3a3c3c89fd5cb3` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:47.191751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e02e5ed-3cbf-4644-a657-5c4102a4a063 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cb0f225034f94be99b3a3c3c89fd5cb3)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:47.192159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c256c47-7170-49d3-b658-241cbf175eff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:47.192866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c256c47-7170-49d3-b658-241cbf175eff admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:47.193144 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c256c47-7170-49d3-b658-241cbf175eff admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:47.193415 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c256c47-7170-49d3-b658-241cbf175eff admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:47.193653 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e02e5ed-3cbf-4644-a657-5c4102a4a063 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:47.194041 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c256c47-7170-49d3-b658-241cbf175eff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:47.195964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c256c47-7170-49d3-b658-241cbf175eff admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:47.290369 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1356/2696] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:09:47 2022] POST /identity/v3/projects => generated 377 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:09:47.294087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c256c47-7170-49d3-b658-241cbf175eff admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:47.295701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-4c256c47-7170-49d3-b658-241cbf175eff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:47.360794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c256c47-7170-49d3-b658-241cbf175eff admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:47.365534 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554c6456-ae4a-49e4-8ad4-5949b5c7fc6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:47.366131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554c6456-ae4a-49e4-8ad4-5949b5c7fc6a admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:47.366434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554c6456-ae4a-49e4-8ad4-5949b5c7fc6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:47.366707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554c6456-ae4a-49e4-8ad4-5949b5c7fc6a admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:47.367332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554c6456-ae4a-49e4-8ad4-5949b5c7fc6a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:47.369275 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554c6456-ae4a-49e4-8ad4-5949b5c7fc6a admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:47.396324 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-9e02e5ed-3cbf-4644-a657-5c4102a4a063 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cb0f225034f94be99b3a3c3c89fd5cb3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:47.461498 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-9e02e5ed-3cbf-4644-a657-5c4102a4a063 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cb0f225034f94be99b3a3c3c89fd5cb3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:47.461706 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1357/2697] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:09:47 2022] POST /identity/v3/users => generated 295 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:09:47.462203 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-9e02e5ed-3cbf-4644-a657-5c4102a4a063 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cb0f225034f94be99b3a3c3c89fd5cb3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:47.465840 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554c6456-ae4a-49e4-8ad4-5949b5c7fc6a admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:47.467522 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-554c6456-ae4a-49e4-8ad4-5949b5c7fc6a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:47.471526 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=79,keystone:INSERT=5,keystone:DELETE=16 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:47.478052 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-9e02e5ed-3cbf-4644-a657-5c4102a4a063 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cb0f225034f94be99b3a3c3c89fd5cb3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:47.480163 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554c6456-ae4a-49e4-8ad4-5949b5c7fc6a admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:47.484940 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a44b07d-985a-45ab-b9bd-67e1e5c79262 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:47.485529 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a44b07d-985a-45ab-b9bd-67e1e5c79262 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:47.485717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-9e02e5ed-3cbf-4644-a657-5c4102a4a063 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cb0f225034f94be99b3a3c3c89fd5cb3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:47.485825 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a44b07d-985a-45ab-b9bd-67e1e5c79262 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:47.486061 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a44b07d-985a-45ab-b9bd-67e1e5c79262 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:47.486740 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a44b07d-985a-45ab-b9bd-67e1e5c79262 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:47.488316 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a44b07d-985a-45ab-b9bd-67e1e5c79262 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:47.491978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1341/2698] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:09:47 2022] DELETE /identity/v3/users/cb0f225034f94be99b3a3c3c89fd5cb3 => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:09:47.502270 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1358/2699] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:47 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:09:47.503744 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e02e5ed-3cbf-4644-a657-5c4102a4a063 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:47.566792 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e02e5ed-3cbf-4644-a657-5c4102a4a063 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:47.571417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201cbcf5-0763-4b23-81f5-c51bb4f9e1e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:47.572071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201cbcf5-0763-4b23-81f5-c51bb4f9e1e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:47.572347 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201cbcf5-0763-4b23-81f5-c51bb4f9e1e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:47.572616 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201cbcf5-0763-4b23-81f5-c51bb4f9e1e2 admin admin] PATH_INFO: `/v3/projects/29002de470164088b4659341e49e0ba5/users/3df5ac0ac1024a79af80067d2f977dd1/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:47.587361 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201cbcf5-0763-4b23-81f5-c51bb4f9e1e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=29002de470164088b4659341e49e0ba5, user_id=3df5ac0ac1024a79af80067d2f977dd1, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:47.590032 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201cbcf5-0763-4b23-81f5-c51bb4f9e1e2 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:47.668037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1342/2700] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:09:47 2022] PUT /identity/v3/projects/29002de470164088b4659341e49e0ba5/users/3df5ac0ac1024a79af80067d2f977dd1/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:47.672115 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a44b07d-985a-45ab-b9bd-67e1e5c79262 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:47.682817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a44b07d-985a-45ab-b9bd-67e1e5c79262 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:47.734855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:47.749498 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d70cb3b-e736-4e93-85fe-ea7659730ebb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:47.752714 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d70cb3b-e736-4e93-85fe-ea7659730ebb admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:47.752974 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d70cb3b-e736-4e93-85fe-ea7659730ebb admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:47.753247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d70cb3b-e736-4e93-85fe-ea7659730ebb admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:47.753906 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d70cb3b-e736-4e93-85fe-ea7659730ebb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:47.757512 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d70cb3b-e736-4e93-85fe-ea7659730ebb admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:47.768731 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1359/2701] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:47 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:09:47.772561 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201cbcf5-0763-4b23-81f5-c51bb4f9e1e2 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:47.783169 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201cbcf5-0763-4b23-81f5-c51bb4f9e1e2 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:47.787948 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4604ae-e166-4c87-ba65-e259779ebef7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:47.788607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4604ae-e166-4c87-ba65-e259779ebef7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:47.788885 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4604ae-e166-4c87-ba65-e259779ebef7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:47.789155 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4604ae-e166-4c87-ba65-e259779ebef7 admin admin] PATH_INFO: `/v3/projects/29002de470164088b4659341e49e0ba5/users/3df5ac0ac1024a79af80067d2f977dd1/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:47.791078 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4604ae-e166-4c87-ba65-e259779ebef7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=29002de470164088b4659341e49e0ba5, user_id=3df5ac0ac1024a79af80067d2f977dd1, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:47.793705 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4604ae-e166-4c87-ba65-e259779ebef7 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:47.804512 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1343/2702] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:09:47 2022] PUT /identity/v3/projects/29002de470164088b4659341e49e0ba5/users/3df5ac0ac1024a79af80067d2f977dd1/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:47.832507 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6431eb-f838-4322-b58f-57dbe250b710 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:47.832787 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6431eb-f838-4322-b58f-57dbe250b710 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:47.833077 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc6431eb-f838-4322-b58f-57dbe250b710 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:47.860958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-fc6431eb-f838-4322-b58f-57dbe250b710 None None] MFA Rules not processed for user `3df5ac0ac1024a79af80067d2f977dd1`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:47.862152 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-fc6431eb-f838-4322-b58f-57dbe250b710 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:47.899009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4604ae-e166-4c87-ba65-e259779ebef7 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:47.901973 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ca4604ae-e166-4c87-ba65-e259779ebef7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:47.938590 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1360/2703] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:47 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:47.950237 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6431eb-f838-4322-b58f-57dbe250b710 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:47.951661 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-fc6431eb-f838-4322-b58f-57dbe250b710 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:47.986454 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4604ae-e166-4c87-ba65-e259779ebef7 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:47.991212 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a65a9d-8596-4951-a598-10ff89c20252 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:47.991839 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a65a9d-8596-4951-a598-10ff89c20252 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:47.992118 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a65a9d-8596-4951-a598-10ff89c20252 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:47.992388 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a65a9d-8596-4951-a598-10ff89c20252 admin admin] PATH_INFO: `/v3/projects/cc406d76de034732a1e8320781d0742a` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:47.993515 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a65a9d-8596-4951-a598-10ff89c20252 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cc406d76de034732a1e8320781d0742a)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:47.995590 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a65a9d-8596-4951-a598-10ff89c20252 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:48.009828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc6431eb-f838-4322-b58f-57dbe250b710 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:48.013482 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3c3c4f-d174-4f11-b6f7-e464db05f088 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:48.014084 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3c3c4f-d174-4f11-b6f7-e464db05f088 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:48.014393 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3c3c4f-d174-4f11-b6f7-e464db05f088 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:48.014666 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3c3c4f-d174-4f11-b6f7-e464db05f088 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:48.016992 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-7f3c3c4f-d174-4f11-b6f7-e464db05f088 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:48.088447 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3c3c4f-d174-4f11-b6f7-e464db05f088 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:48.089886 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3c3c4f-d174-4f11-b6f7-e464db05f088 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:48.090680 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-31a65a9d-8596-4951-a598-10ff89c20252 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cc406d76de034732a1e8320781d0742a', 'request_id': 'req-31a65a9d-8596-4951-a598-10ff89c20252'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:48.093339 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=92,keystone:DELETE=5,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:48.099060 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1361/2704] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:47 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:48.169846 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-31a65a9d-8596-4951-a598-10ff89c20252 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cc406d76de034732a1e8320781d0742a', 'request_id': 'req-31a65a9d-8596-4951-a598-10ff89c20252'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:48.177038 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-31a65a9d-8596-4951-a598-10ff89c20252 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cc406d76de034732a1e8320781d0742a', 'request_id': 'req-31a65a9d-8596-4951-a598-10ff89c20252'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:48.190233 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-31a65a9d-8596-4951-a598-10ff89c20252 admin admin] The token cache is being invalidate because project cc406d76de034732a1e8320781d0742a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:09:48.195026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-31a65a9d-8596-4951-a598-10ff89c20252 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:48.195026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-31a65a9d-8596-4951-a598-10ff89c20252 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:48.195026 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1344/2705] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:09:47 2022] DELETE /identity/v3/projects/cc406d76de034732a1e8320781d0742a => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:09:48.197645 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5b41ba-589a-4a32-b73a-1026b6fbab30 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:48.197925 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5b41ba-589a-4a32-b73a-1026b6fbab30 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:48.198194 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d5b41ba-589a-4a32-b73a-1026b6fbab30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:48.218829 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-8d5b41ba-589a-4a32-b73a-1026b6fbab30 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:48.220199 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8d5b41ba-589a-4a32-b73a-1026b6fbab30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:48.239918 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=100 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:48.280559 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1362/2706] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:09:48.288600 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6885070a-b442-40af-a96c-f873aa297cb5 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:48.288887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6885070a-b442-40af-a96c-f873aa297cb5 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:48.289157 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6885070a-b442-40af-a96c-f873aa297cb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:48.310008 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-6885070a-b442-40af-a96c-f873aa297cb5 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:48.311345 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-6885070a-b442-40af-a96c-f873aa297cb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:48.320070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1345/2707] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:48.324445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5b41ba-589a-4a32-b73a-1026b6fbab30 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:48.335501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d5b41ba-589a-4a32-b73a-1026b6fbab30 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:48.340255 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd063dda-c2b3-4787-a3e5-fea6fec5b4c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:48.340855 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd063dda-c2b3-4787-a3e5-fea6fec5b4c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:48.341129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd063dda-c2b3-4787-a3e5-fea6fec5b4c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:48.341393 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd063dda-c2b3-4787-a3e5-fea6fec5b4c7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:48.342088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd063dda-c2b3-4787-a3e5-fea6fec5b4c7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:09:48.342411 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd063dda-c2b3-4787-a3e5-fea6fec5b4c7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:48.344096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd063dda-c2b3-4787-a3e5-fea6fec5b4c7 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:48.355770 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1363/2708] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:09:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:09:48.360790 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6885070a-b442-40af-a96c-f873aa297cb5 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:48.369939 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6885070a-b442-40af-a96c-f873aa297cb5 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:48.375155 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca08cf82-bf4d-4f41-a992-9546e7f1c290 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:48.375741 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca08cf82-bf4d-4f41-a992-9546e7f1c290 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:48.376024 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca08cf82-bf4d-4f41-a992-9546e7f1c290 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:48.376289 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca08cf82-bf4d-4f41-a992-9546e7f1c290 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:48.376905 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca08cf82-bf4d-4f41-a992-9546e7f1c290 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:48.378810 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca08cf82-bf4d-4f41-a992-9546e7f1c290 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:48.474988 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1346/2709] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:09:48 2022] POST /identity/v3/projects => generated 363 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:09:48.479370 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd063dda-c2b3-4787-a3e5-fea6fec5b4c7 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:48.490304 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd063dda-c2b3-4787-a3e5-fea6fec5b4c7 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:48.549765 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61161226-344b-4cfd-bee2-aca0313a7c56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:48.550391 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61161226-344b-4cfd-bee2-aca0313a7c56 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:48.550671 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61161226-344b-4cfd-bee2-aca0313a7c56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:48.550941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61161226-344b-4cfd-bee2-aca0313a7c56 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:48.551575 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61161226-344b-4cfd-bee2-aca0313a7c56 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:48.553480 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61161226-344b-4cfd-bee2-aca0313a7c56 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:48.633101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1364/2710] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:09:48 2022] POST /identity/v3/users => generated 282 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:09:48.637410 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca08cf82-bf4d-4f41-a992-9546e7f1c290 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:48.647784 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca08cf82-bf4d-4f41-a992-9546e7f1c290 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:48.652570 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0b5629-4c33-4d49-9e85-8dec2d24ef19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:48.653164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0b5629-4c33-4d49-9e85-8dec2d24ef19 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:48.657269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0b5629-4c33-4d49-9e85-8dec2d24ef19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:48.657269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc0b5629-4c33-4d49-9e85-8dec2d24ef19 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:48.657269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0b5629-4c33-4d49-9e85-8dec2d24ef19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:48.657269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc0b5629-4c33-4d49-9e85-8dec2d24ef19 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:48.678075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1347/2711] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:48 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:09:48.680331 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61161226-344b-4cfd-bee2-aca0313a7c56 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:48.691488 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61161226-344b-4cfd-bee2-aca0313a7c56 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:48.696300 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3c8581-5269-4b3d-8598-1ef2b6e1c013 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:48.696904 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3c8581-5269-4b3d-8598-1ef2b6e1c013 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:48.697186 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3c8581-5269-4b3d-8598-1ef2b6e1c013 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:48.697466 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3c8581-5269-4b3d-8598-1ef2b6e1c013 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:48.698140 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3c8581-5269-4b3d-8598-1ef2b6e1c013 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:48.699795 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3c8581-5269-4b3d-8598-1ef2b6e1c013 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:48.712944 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1365/2712] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:09:48 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:09:48.717137 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0b5629-4c33-4d49-9e85-8dec2d24ef19 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:48.728134 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc0b5629-4c33-4d49-9e85-8dec2d24ef19 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:48.733182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7aa988-5860-4eb2-a316-5aef12a4ccbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:48.733844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7aa988-5860-4eb2-a316-5aef12a4ccbf admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:48.737638 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7aa988-5860-4eb2-a316-5aef12a4ccbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:48.737638 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7aa988-5860-4eb2-a316-5aef12a4ccbf admin admin] PATH_INFO: `/v3/projects/c36dff07dcb04ee29210ebac9eb03804/users/178e5adbb5634ccdbba91ace9eac23fb/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:48.751322 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7aa988-5860-4eb2-a316-5aef12a4ccbf admin admin] RBAC: Authorizing `identity:create_grant(project_id=c36dff07dcb04ee29210ebac9eb03804, user_id=178e5adbb5634ccdbba91ace9eac23fb, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:48.754182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7aa988-5860-4eb2-a316-5aef12a4ccbf admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:48.832167 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1348/2713] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:09:48 2022] PUT /identity/v3/projects/c36dff07dcb04ee29210ebac9eb03804/users/178e5adbb5634ccdbba91ace9eac23fb/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:48.842757 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d536e11-4fb3-494f-b3ba-d9099026ae1d None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:48.843040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d536e11-4fb3-494f-b3ba-d9099026ae1d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:48.843316 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d536e11-4fb3-494f-b3ba-d9099026ae1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:48.871246 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-2d536e11-4fb3-494f-b3ba-d9099026ae1d None None] MFA Rules not processed for user `178e5adbb5634ccdbba91ace9eac23fb`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:48.872502 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-2d536e11-4fb3-494f-b3ba-d9099026ae1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:48.938028 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1366/2714] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:48 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:48.949543 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7aa988-5860-4eb2-a316-5aef12a4ccbf admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:48.951333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-1d7aa988-5860-4eb2-a316-5aef12a4ccbf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:49.009500 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7aa988-5860-4eb2-a316-5aef12a4ccbf admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:49.013127 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc3bebd-d4da-4516-941e-801cb0973261 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:49.013724 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc3bebd-d4da-4516-941e-801cb0973261 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:49.014002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc3bebd-d4da-4516-941e-801cb0973261 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:49.014272 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc3bebd-d4da-4516-941e-801cb0973261 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:49.022668 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc3bebd-d4da-4516-941e-801cb0973261 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:49.024147 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc3bebd-d4da-4516-941e-801cb0973261 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:49.033355 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1349/2715] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:48 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:50.032342 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d536e11-4fb3-494f-b3ba-d9099026ae1d None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:50.033764 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-2d536e11-4fb3-494f-b3ba-d9099026ae1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:50.047406 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d536e11-4fb3-494f-b3ba-d9099026ae1d None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:50.051085 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d33f76-94ba-4aae-ad57-a72547e27744 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:50.051590 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d33f76-94ba-4aae-ad57-a72547e27744 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:50.051864 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d33f76-94ba-4aae-ad57-a72547e27744 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:50.052132 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d33f76-94ba-4aae-ad57-a72547e27744 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:50.053907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-40d33f76-94ba-4aae-ad57-a72547e27744 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:50.072802 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:50.118578 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d33f76-94ba-4aae-ad57-a72547e27744 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:50.120346 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d33f76-94ba-4aae-ad57-a72547e27744 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:50.130821 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1367/2716] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:50 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:51.632706 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc3bebd-d4da-4516-941e-801cb0973261 service cinder] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:51.642589 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc3bebd-d4da-4516-941e-801cb0973261 service cinder] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:51.646439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59c24131-a5ee-4a35-ad17-3b497eb75dcd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:51.647035 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c24131-a5ee-4a35-ad17-3b497eb75dcd service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:51.647312 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c24131-a5ee-4a35-ad17-3b497eb75dcd service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:51.647584 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c24131-a5ee-4a35-ad17-3b497eb75dcd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:51.649331 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-59c24131-a5ee-4a35-ad17-3b497eb75dcd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:51.717547 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c24131-a5ee-4a35-ad17-3b497eb75dcd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:51.719133 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59c24131-a5ee-4a35-ad17-3b497eb75dcd service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:51.728557 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1350/2717] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:51 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:56.496411 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d33f76-94ba-4aae-ad57-a72547e27744 service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:56.497846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-40d33f76-94ba-4aae-ad57-a72547e27744 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:56.561994 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d33f76-94ba-4aae-ad57-a72547e27744 service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:56.567129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a17004c-d006-4c51-ac78-d6e1bb5d7477 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:56.567738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a17004c-d006-4c51-ac78-d6e1bb5d7477 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:56.568012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a17004c-d006-4c51-ac78-d6e1bb5d7477 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:56.568280 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a17004c-d006-4c51-ac78-d6e1bb5d7477 admin admin] PATH_INFO: `/v3/users/edd988c9077142d6bdf6e93f16a1ad8e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:56.569371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a17004c-d006-4c51-ac78-d6e1bb5d7477 admin admin] RBAC: Authorizing `identity:delete_user(user_id=edd988c9077142d6bdf6e93f16a1ad8e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:56.571295 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a17004c-d006-4c51-ac78-d6e1bb5d7477 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:56.771877 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7a17004c-d006-4c51-ac78-d6e1bb5d7477 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'edd988c9077142d6bdf6e93f16a1ad8e'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:56.831296 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7a17004c-d006-4c51-ac78-d6e1bb5d7477 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'edd988c9077142d6bdf6e93f16a1ad8e'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:56.843698 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7a17004c-d006-4c51-ac78-d6e1bb5d7477 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'edd988c9077142d6bdf6e93f16a1ad8e'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:56.844338 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7a17004c-d006-4c51-ac78-d6e1bb5d7477 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'edd988c9077142d6bdf6e93f16a1ad8e'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:56.851996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7a17004c-d006-4c51-ac78-d6e1bb5d7477 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'edd988c9077142d6bdf6e93f16a1ad8e'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:56.857834 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1368/2718] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:09:56 2022] DELETE /identity/v3/users/edd988c9077142d6bdf6e93f16a1ad8e => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:56.863165 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74d3836-6a80-4e30-9549-4e492e8c4f7f None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:56.863676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74d3836-6a80-4e30-9549-4e492e8c4f7f None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:56.863778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74d3836-6a80-4e30-9549-4e492e8c4f7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:56.882430 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-f74d3836-6a80-4e30-9549-4e492e8c4f7f None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:09:56.883792 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f74d3836-6a80-4e30-9549-4e492e8c4f7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:56.891588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:DELETE=3,keystone:SELECT=95,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:56.943206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1351/2719] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:09:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:09:56.953176 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a17004c-d006-4c51-ac78-d6e1bb5d7477 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:56.954609 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-7a17004c-d006-4c51-ac78-d6e1bb5d7477 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:56.997887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:57.013093 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a17004c-d006-4c51-ac78-d6e1bb5d7477 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:57.017603 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb73ef9-6843-4f9d-bacc-16616725f22c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:57.018213 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb73ef9-6843-4f9d-bacc-16616725f22c service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:57.018522 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb73ef9-6843-4f9d-bacc-16616725f22c service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:57.018793 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb73ef9-6843-4f9d-bacc-16616725f22c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:57.033051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb73ef9-6843-4f9d-bacc-16616725f22c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:57.033051 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb73ef9-6843-4f9d-bacc-16616725f22c service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:57.041423 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1369/2720] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:09:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:09:57.451442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74d3836-6a80-4e30-9549-4e492e8c4f7f None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:57.459594 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f74d3836-6a80-4e30-9549-4e492e8c4f7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:57.465552 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74d3836-6a80-4e30-9549-4e492e8c4f7f None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:57.470518 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e282fc3-dcea-4385-b21f-ebf8740aabd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:57.471197 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e282fc3-dcea-4385-b21f-ebf8740aabd4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:57.471495 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e282fc3-dcea-4385-b21f-ebf8740aabd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:57.471768 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e282fc3-dcea-4385-b21f-ebf8740aabd4 admin admin] PATH_INFO: `/v3/projects/e56e32789db6408db7088a5c6b387267` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:57.472978 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e282fc3-dcea-4385-b21f-ebf8740aabd4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e56e32789db6408db7088a5c6b387267)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:57.474947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e282fc3-dcea-4385-b21f-ebf8740aabd4 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:57.576618 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2e282fc3-dcea-4385-b21f-ebf8740aabd4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e56e32789db6408db7088a5c6b387267', 'request_id': 'req-2e282fc3-dcea-4385-b21f-ebf8740aabd4'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:57.649599 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2e282fc3-dcea-4385-b21f-ebf8740aabd4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e56e32789db6408db7088a5c6b387267', 'request_id': 'req-2e282fc3-dcea-4385-b21f-ebf8740aabd4'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:57.655840 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2e282fc3-dcea-4385-b21f-ebf8740aabd4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e56e32789db6408db7088a5c6b387267', 'request_id': 'req-2e282fc3-dcea-4385-b21f-ebf8740aabd4'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:57.667247 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2e282fc3-dcea-4385-b21f-ebf8740aabd4 admin admin] The token cache is being invalidate because project e56e32789db6408db7088a5c6b387267 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:09:57.667562 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2e282fc3-dcea-4385-b21f-ebf8740aabd4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:57.668184 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-2e282fc3-dcea-4385-b21f-ebf8740aabd4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:57.669653 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1352/2721] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:09:57 2022] DELETE /identity/v3/projects/e56e32789db6408db7088a5c6b387267 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:57.674054 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb73ef9-6843-4f9d-bacc-16616725f22c service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:57.684032 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb73ef9-6843-4f9d-bacc-16616725f22c service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:57.752230 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7ce3b3-d257-42f5-91b9-c0b272549de0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:57.752869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7ce3b3-d257-42f5-91b9-c0b272549de0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:57.753229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7ce3b3-d257-42f5-91b9-c0b272549de0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:57.753498 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7ce3b3-d257-42f5-91b9-c0b272549de0 admin admin] PATH_INFO: `/v3/users/47a7beead42f450a9e1840c9e9563164` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:57.754688 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7ce3b3-d257-42f5-91b9-c0b272549de0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=47a7beead42f450a9e1840c9e9563164)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:57.756679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7ce3b3-d257-42f5-91b9-c0b272549de0 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:58.046027 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-cb7ce3b3-d257-42f5-91b9-c0b272549de0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '47a7beead42f450a9e1840c9e9563164'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:58.123217 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-cb7ce3b3-d257-42f5-91b9-c0b272549de0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '47a7beead42f450a9e1840c9e9563164'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:58.138110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-cb7ce3b3-d257-42f5-91b9-c0b272549de0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '47a7beead42f450a9e1840c9e9563164'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:58.138768 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-cb7ce3b3-d257-42f5-91b9-c0b272549de0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '47a7beead42f450a9e1840c9e9563164'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:58.145875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-cb7ce3b3-d257-42f5-91b9-c0b272549de0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '47a7beead42f450a9e1840c9e9563164'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:58.152102 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1370/2722] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:09:57 2022] DELETE /identity/v3/users/47a7beead42f450a9e1840c9e9563164 => generated 0 bytes in 478 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:58.587780 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e282fc3-dcea-4385-b21f-ebf8740aabd4 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:58.589267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2e282fc3-dcea-4385-b21f-ebf8740aabd4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:58.604135 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e282fc3-dcea-4385-b21f-ebf8740aabd4 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:58.661829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c5c803-ebf0-4c1d-8f68-60c3e742b391 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:58.662437 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c5c803-ebf0-4c1d-8f68-60c3e742b391 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:58.662712 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c5c803-ebf0-4c1d-8f68-60c3e742b391 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:58.662977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c5c803-ebf0-4c1d-8f68-60c3e742b391 admin admin] PATH_INFO: `/v3/projects/d8e428a1de6f4533a4bc50d7a4a8894f` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:58.664087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c5c803-ebf0-4c1d-8f68-60c3e742b391 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d8e428a1de6f4533a4bc50d7a4a8894f)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:58.666123 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c5c803-ebf0-4c1d-8f68-60c3e742b391 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:58.803705 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:58.810818 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a6c5c803-ebf0-4c1d-8f68-60c3e742b391 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd8e428a1de6f4533a4bc50d7a4a8894f', 'request_id': 'req-a6c5c803-ebf0-4c1d-8f68-60c3e742b391'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:58.924198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a6c5c803-ebf0-4c1d-8f68-60c3e742b391 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd8e428a1de6f4533a4bc50d7a4a8894f', 'request_id': 'req-a6c5c803-ebf0-4c1d-8f68-60c3e742b391'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:58.937872 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a6c5c803-ebf0-4c1d-8f68-60c3e742b391 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd8e428a1de6f4533a4bc50d7a4a8894f', 'request_id': 'req-a6c5c803-ebf0-4c1d-8f68-60c3e742b391'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:58.950128 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a6c5c803-ebf0-4c1d-8f68-60c3e742b391 admin admin] The token cache is being invalidate because project d8e428a1de6f4533a4bc50d7a4a8894f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:09:58.950530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a6c5c803-ebf0-4c1d-8f68-60c3e742b391 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:58.951225 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a6c5c803-ebf0-4c1d-8f68-60c3e742b391 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:58.952690 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1353/2723] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:09:58 2022] DELETE /identity/v3/projects/d8e428a1de6f4533a4bc50d7a4a8894f => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:58.957210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7ce3b3-d257-42f5-91b9-c0b272549de0 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:58.958626 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-cb7ce3b3-d257-42f5-91b9-c0b272549de0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:58.971106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7ce3b3-d257-42f5-91b9-c0b272549de0 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:59.025832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e149de8-78a6-4012-b3e8-717f9c616cc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:59.030833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e149de8-78a6-4012-b3e8-717f9c616cc2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:59.030833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e149de8-78a6-4012-b3e8-717f9c616cc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:59.030833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e149de8-78a6-4012-b3e8-717f9c616cc2 admin admin] PATH_INFO: `/v3/users/3df5ac0ac1024a79af80067d2f977dd1` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:59.030833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e149de8-78a6-4012-b3e8-717f9c616cc2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3df5ac0ac1024a79af80067d2f977dd1)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:59.030833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e149de8-78a6-4012-b3e8-717f9c616cc2 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:59.031585 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:09:59.252447 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-3e149de8-78a6-4012-b3e8-717f9c616cc2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3df5ac0ac1024a79af80067d2f977dd1'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:59.306524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-3e149de8-78a6-4012-b3e8-717f9c616cc2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3df5ac0ac1024a79af80067d2f977dd1'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:59.319480 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-3e149de8-78a6-4012-b3e8-717f9c616cc2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3df5ac0ac1024a79af80067d2f977dd1'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:59.320105 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-3e149de8-78a6-4012-b3e8-717f9c616cc2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3df5ac0ac1024a79af80067d2f977dd1'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:59.327017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-3e149de8-78a6-4012-b3e8-717f9c616cc2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3df5ac0ac1024a79af80067d2f977dd1'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:09:59.333326 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1371/2724] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:09:58 2022] DELETE /identity/v3/users/3df5ac0ac1024a79af80067d2f977dd1 => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:09:59.726711 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c5c803-ebf0-4c1d-8f68-60c3e742b391 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:09:59.729006 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a6c5c803-ebf0-4c1d-8f68-60c3e742b391 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:09:59.742439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c5c803-ebf0-4c1d-8f68-60c3e742b391 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:09:59.799668 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f7a8369-6230-47db-afbe-0c97d7bed32b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:09:59.800326 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7a8369-6230-47db-afbe-0c97d7bed32b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:09:59.800601 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7a8369-6230-47db-afbe-0c97d7bed32b admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:09:59.800870 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f7a8369-6230-47db-afbe-0c97d7bed32b admin admin] PATH_INFO: `/v3/projects/29002de470164088b4659341e49e0ba5` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:09:59.801976 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7a8369-6230-47db-afbe-0c97d7bed32b admin admin] RBAC: Authorizing `identity:delete_project(project_id=29002de470164088b4659341e49e0ba5)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:09:59.803906 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f7a8369-6230-47db-afbe-0c97d7bed32b admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:09:59.930995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-6f7a8369-6230-47db-afbe-0c97d7bed32b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '29002de470164088b4659341e49e0ba5', 'request_id': 'req-6f7a8369-6230-47db-afbe-0c97d7bed32b'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:00.014022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-6f7a8369-6230-47db-afbe-0c97d7bed32b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '29002de470164088b4659341e49e0ba5', 'request_id': 'req-6f7a8369-6230-47db-afbe-0c97d7bed32b'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:00.020858 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-6f7a8369-6230-47db-afbe-0c97d7bed32b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '29002de470164088b4659341e49e0ba5', 'request_id': 'req-6f7a8369-6230-47db-afbe-0c97d7bed32b'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:00.033500 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-6f7a8369-6230-47db-afbe-0c97d7bed32b admin admin] The token cache is being invalidate because project 29002de470164088b4659341e49e0ba5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:10:00.033790 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-6f7a8369-6230-47db-afbe-0c97d7bed32b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:00.034472 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-6f7a8369-6230-47db-afbe-0c97d7bed32b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:00.035947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1354/2725] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:09:59 2022] DELETE /identity/v3/projects/29002de470164088b4659341e49e0ba5 => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:10:00.040946 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fead094-c894-41c5-983a-ab1a1ab4de7f None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:00.041418 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fead094-c894-41c5-983a-ab1a1ab4de7f None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:00.041526 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fead094-c894-41c5-983a-ab1a1ab4de7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:00.061449 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-9fead094-c894-41c5-983a-ab1a1ab4de7f None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:10:00.062691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-9fead094-c894-41c5-983a-ab1a1ab4de7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:00.120879 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1372/2726] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:10:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:10:00.128883 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16379af-6404-489d-9c98-e5de502c348e None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:00.129163 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16379af-6404-489d-9c98-e5de502c348e None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:00.129436 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16379af-6404-489d-9c98-e5de502c348e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:00.149522 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-e16379af-6404-489d-9c98-e5de502c348e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:10:00.150748 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e16379af-6404-489d-9c98-e5de502c348e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:00.159192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1355/2727] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:10:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:10:00.163345 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fead094-c894-41c5-983a-ab1a1ab4de7f None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:00.173028 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fead094-c894-41c5-983a-ab1a1ab4de7f None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:00.177737 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453f5efe-74a7-4646-b3f2-bfce46b91e58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:00.178343 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453f5efe-74a7-4646-b3f2-bfce46b91e58 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:00.178632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453f5efe-74a7-4646-b3f2-bfce46b91e58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:00.178902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453f5efe-74a7-4646-b3f2-bfce46b91e58 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:00.179546 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453f5efe-74a7-4646-b3f2-bfce46b91e58 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:10:00.179848 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453f5efe-74a7-4646-b3f2-bfce46b91e58 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:00.181434 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453f5efe-74a7-4646-b3f2-bfce46b91e58 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:00.193132 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1373/2728] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:10:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:10:00.197262 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16379af-6404-489d-9c98-e5de502c348e None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:00.206893 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16379af-6404-489d-9c98-e5de502c348e None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:00.211509 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99da98a0-da9e-4281-8f28-bb6e4366c2a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:00.212103 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99da98a0-da9e-4281-8f28-bb6e4366c2a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:00.212378 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99da98a0-da9e-4281-8f28-bb6e4366c2a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:00.212645 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99da98a0-da9e-4281-8f28-bb6e4366c2a2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:00.213267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99da98a0-da9e-4281-8f28-bb6e4366c2a2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:00.215213 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99da98a0-da9e-4281-8f28-bb6e4366c2a2 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:00.286107 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1356/2729] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:10:00 2022] POST /identity/v3/projects => generated 371 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:10:00.290356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453f5efe-74a7-4646-b3f2-bfce46b91e58 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:00.300440 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453f5efe-74a7-4646-b3f2-bfce46b91e58 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:00.356751 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceac42b1-b94e-43ff-8ba0-22520451d6e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:00.357397 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceac42b1-b94e-43ff-8ba0-22520451d6e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:00.357668 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceac42b1-b94e-43ff-8ba0-22520451d6e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:00.357941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceac42b1-b94e-43ff-8ba0-22520451d6e4 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:00.358612 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceac42b1-b94e-43ff-8ba0-22520451d6e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:00.360663 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceac42b1-b94e-43ff-8ba0-22520451d6e4 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:00.371436 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:10:00.455888 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1374/2730] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:10:00 2022] POST /identity/v3/users => generated 286 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:10:00.460222 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99da98a0-da9e-4281-8f28-bb6e4366c2a2 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:00.473078 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99da98a0-da9e-4281-8f28-bb6e4366c2a2 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:00.476121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752f3d8c-c3da-44b1-8766-35b6f6c0509d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:00.476811 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752f3d8c-c3da-44b1-8766-35b6f6c0509d admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:00.477021 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752f3d8c-c3da-44b1-8766-35b6f6c0509d admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:00.477292 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752f3d8c-c3da-44b1-8766-35b6f6c0509d admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:00.477931 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752f3d8c-c3da-44b1-8766-35b6f6c0509d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:00.479582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752f3d8c-c3da-44b1-8766-35b6f6c0509d admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:00.489996 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1357/2731] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:10:00 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:10:00.495660 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceac42b1-b94e-43ff-8ba0-22520451d6e4 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:00.504813 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceac42b1-b94e-43ff-8ba0-22520451d6e4 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:00.510584 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ff1f1a-6dcd-4c3e-82f9-a1464f59759b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:00.511233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ff1f1a-6dcd-4c3e-82f9-a1464f59759b admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:00.511530 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ff1f1a-6dcd-4c3e-82f9-a1464f59759b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:00.511800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ff1f1a-6dcd-4c3e-82f9-a1464f59759b admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:00.512508 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ff1f1a-6dcd-4c3e-82f9-a1464f59759b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:00.514169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ff1f1a-6dcd-4c3e-82f9-a1464f59759b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:00.526120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1375/2732] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:10:00 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:10:00.530516 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752f3d8c-c3da-44b1-8766-35b6f6c0509d admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:00.541297 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752f3d8c-c3da-44b1-8766-35b6f6c0509d admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:00.545985 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1617cc-eb56-455f-b0d7-993fc7f81d3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:00.546675 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1617cc-eb56-455f-b0d7-993fc7f81d3b admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:00.546973 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1617cc-eb56-455f-b0d7-993fc7f81d3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:00.547852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b1617cc-eb56-455f-b0d7-993fc7f81d3b admin admin] PATH_INFO: `/v3/projects/c4ce494070ca4b76a93a6a4a9f42414c/users/256d2963df4c4e48876583485de59674/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:00.562286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1617cc-eb56-455f-b0d7-993fc7f81d3b admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4ce494070ca4b76a93a6a4a9f42414c, user_id=256d2963df4c4e48876583485de59674, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:00.565593 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b1617cc-eb56-455f-b0d7-993fc7f81d3b admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:00.567780 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:10:00.665377 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1358/2733] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:10:00 2022] PUT /identity/v3/projects/c4ce494070ca4b76a93a6a4a9f42414c/users/256d2963df4c4e48876583485de59674/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:10:00.670630 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bbbb6e-7ec8-4ae2-a727-1b3a358deda6 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:00.670911 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bbbb6e-7ec8-4ae2-a727-1b3a358deda6 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:00.671207 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bbbb6e-7ec8-4ae2-a727-1b3a358deda6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:00.690680 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-12bbbb6e-7ec8-4ae2-a727-1b3a358deda6 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:10:00.697147 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-12bbbb6e-7ec8-4ae2-a727-1b3a358deda6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:00.750022 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1376/2734] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:10:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:10:00.760377 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1617cc-eb56-455f-b0d7-993fc7f81d3b admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:00.762105 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-0b1617cc-eb56-455f-b0d7-993fc7f81d3b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:00.819539 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b1617cc-eb56-455f-b0d7-993fc7f81d3b admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:00.823275 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2891ed-734a-4cf2-a049-810844731531 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:00.823872 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2891ed-734a-4cf2-a049-810844731531 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:00.824173 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2891ed-734a-4cf2-a049-810844731531 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:00.824446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2891ed-734a-4cf2-a049-810844731531 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:00.833816 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2891ed-734a-4cf2-a049-810844731531 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:00.835243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e2891ed-734a-4cf2-a049-810844731531 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:00.844600 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1359/2735] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:10:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:10:02.206093 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba7ef8c-685d-483c-b720-94ed3d158c46 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:02.206614 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba7ef8c-685d-483c-b720-94ed3d158c46 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:02.207496 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba7ef8c-685d-483c-b720-94ed3d158c46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:02.239242 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-8ba7ef8c-685d-483c-b720-94ed3d158c46 None None] MFA Rules not processed for user `256d2963df4c4e48876583485de59674`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:10:02.240460 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-8ba7ef8c-685d-483c-b720-94ed3d158c46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:02.319233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1377/2736] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:10:02 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:10:02.330105 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2891ed-734a-4cf2-a049-810844731531 service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:02.331640 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8e2891ed-734a-4cf2-a049-810844731531 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:02.390891 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e2891ed-734a-4cf2-a049-810844731531 service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:02.395138 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca873b0b-a514-47eb-89e1-3cfab7128d38 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:02.395738 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca873b0b-a514-47eb-89e1-3cfab7128d38 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:02.396018 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca873b0b-a514-47eb-89e1-3cfab7128d38 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:02.396405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca873b0b-a514-47eb-89e1-3cfab7128d38 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:02.404549 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca873b0b-a514-47eb-89e1-3cfab7128d38 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:02.405972 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca873b0b-a514-47eb-89e1-3cfab7128d38 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:02.415602 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1360/2737] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:10:02 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:10:04.400966 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2815091-6af5-46d6-8861-855eb15f35ad None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:04.401479 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2815091-6af5-46d6-8861-855eb15f35ad None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:04.401596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2815091-6af5-46d6-8861-855eb15f35ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:04.422620 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-e2815091-6af5-46d6-8861-855eb15f35ad None None] MFA Rules not processed for user `256d2963df4c4e48876583485de59674`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:10:04.423835 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-e2815091-6af5-46d6-8861-855eb15f35ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:04.433576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1378/2738] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:10:04 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:10:12.411917 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:INSERT=1,keystone:SELECT=54 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:10:14.416851 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=69 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:10:19.638069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca873b0b-a514-47eb-89e1-3cfab7128d38 service cinder] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:19.639485 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ca873b0b-a514-47eb-89e1-3cfab7128d38 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:19.681058 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca873b0b-a514-47eb-89e1-3cfab7128d38 service cinder] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:19.684960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ec2be9-77c8-4801-acaa-169277cc2d89 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:19.685551 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ec2be9-77c8-4801-acaa-169277cc2d89 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:19.685829 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ec2be9-77c8-4801-acaa-169277cc2d89 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:19.686103 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ec2be9-77c8-4801-acaa-169277cc2d89 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:19.687845 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d4ec2be9-77c8-4801-acaa-169277cc2d89 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:19.741593 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ec2be9-77c8-4801-acaa-169277cc2d89 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:19.743039 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ec2be9-77c8-4801-acaa-169277cc2d89 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:19.769450 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1361/2739] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:10:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:10:23.588600 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1debe3-4b56-43ea-9703-16e136231cc2 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:23.588600 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1debe3-4b56-43ea-9703-16e136231cc2 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:23.588600 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1debe3-4b56-43ea-9703-16e136231cc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:23.612667 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-3c1debe3-4b56-43ea-9703-16e136231cc2 None None] MFA Rules not processed for user `256d2963df4c4e48876583485de59674`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:10:23.613897 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-3c1debe3-4b56-43ea-9703-16e136231cc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:23.624099 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1379/2740] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:10:23 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:10:23.800135 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ec2be9-77c8-4801-acaa-169277cc2d89 service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:23.802736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d4ec2be9-77c8-4801-acaa-169277cc2d89 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:23.833817 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ec2be9-77c8-4801-acaa-169277cc2d89 service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:23.843157 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45254c3-273b-4dd7-b1dd-116407f9f654 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:23.844412 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45254c3-273b-4dd7-b1dd-116407f9f654 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:23.845023 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45254c3-273b-4dd7-b1dd-116407f9f654 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:23.846286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45254c3-273b-4dd7-b1dd-116407f9f654 admin admin] PATH_INFO: `/v3/users/d777c43fe5c142b3b5ea5e2485f4e49f` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:23.849906 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45254c3-273b-4dd7-b1dd-116407f9f654 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d777c43fe5c142b3b5ea5e2485f4e49f)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:23.852714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45254c3-273b-4dd7-b1dd-116407f9f654 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:24.148628 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-b45254c3-273b-4dd7-b1dd-116407f9f654 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd777c43fe5c142b3b5ea5e2485f4e49f'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:24.204775 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-b45254c3-273b-4dd7-b1dd-116407f9f654 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd777c43fe5c142b3b5ea5e2485f4e49f'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:24.205484 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-b45254c3-273b-4dd7-b1dd-116407f9f654 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd777c43fe5c142b3b5ea5e2485f4e49f'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:24.223398 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-b45254c3-273b-4dd7-b1dd-116407f9f654 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd777c43fe5c142b3b5ea5e2485f4e49f'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:24.230886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-b45254c3-273b-4dd7-b1dd-116407f9f654 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd777c43fe5c142b3b5ea5e2485f4e49f'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:24.237174 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1362/2741] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:10:23 2022] DELETE /identity/v3/users/d777c43fe5c142b3b5ea5e2485f4e49f => generated 0 bytes in 437 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:10:24.564002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1debe3-4b56-43ea-9703-16e136231cc2 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:24.578306 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1debe3-4b56-43ea-9703-16e136231cc2 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:24.639179 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c77a3f-6f46-42a6-b64b-725b0e07c195 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:24.639827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c77a3f-6f46-42a6-b64b-725b0e07c195 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:24.640097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c77a3f-6f46-42a6-b64b-725b0e07c195 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:24.640354 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c77a3f-6f46-42a6-b64b-725b0e07c195 admin admin] PATH_INFO: `/v3/projects/c21d140d48ba4014bc470e8cf720785d` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:24.641478 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c77a3f-6f46-42a6-b64b-725b0e07c195 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c21d140d48ba4014bc470e8cf720785d)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:24.643447 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c77a3f-6f46-42a6-b64b-725b0e07c195 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:24.777620 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-08c77a3f-6f46-42a6-b64b-725b0e07c195 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c21d140d48ba4014bc470e8cf720785d', 'request_id': 'req-08c77a3f-6f46-42a6-b64b-725b0e07c195'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:24.838501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-08c77a3f-6f46-42a6-b64b-725b0e07c195 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c21d140d48ba4014bc470e8cf720785d', 'request_id': 'req-08c77a3f-6f46-42a6-b64b-725b0e07c195'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:24.844749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-08c77a3f-6f46-42a6-b64b-725b0e07c195 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c21d140d48ba4014bc470e8cf720785d', 'request_id': 'req-08c77a3f-6f46-42a6-b64b-725b0e07c195'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:24.856100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-08c77a3f-6f46-42a6-b64b-725b0e07c195 admin admin] The token cache is being invalidate because project c21d140d48ba4014bc470e8cf720785d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:10:24.856382 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-08c77a3f-6f46-42a6-b64b-725b0e07c195 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:24.857063 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-08c77a3f-6f46-42a6-b64b-725b0e07c195 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:24.858649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1380/2742] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:10:24 2022] DELETE /identity/v3/projects/c21d140d48ba4014bc470e8cf720785d => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:10:24.863608 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befd8f76-ae4e-4071-9d18-f70d38b9b84e None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:24.864107 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befd8f76-ae4e-4071-9d18-f70d38b9b84e None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:24.864215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befd8f76-ae4e-4071-9d18-f70d38b9b84e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:24.921830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-befd8f76-ae4e-4071-9d18-f70d38b9b84e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:10:24.923087 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-befd8f76-ae4e-4071-9d18-f70d38b9b84e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:24.929183 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:10:24.985892 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1363/2743] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:10:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:10:24.992836 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402dd8d7-808b-475d-990b-9a196ca8c299 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:24.993272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402dd8d7-808b-475d-990b-9a196ca8c299 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:24.993550 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402dd8d7-808b-475d-990b-9a196ca8c299 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:25.014150 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-402dd8d7-808b-475d-990b-9a196ca8c299 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:10:25.015390 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-402dd8d7-808b-475d-990b-9a196ca8c299 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:25.023409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1381/2744] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:10:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:10:25.027695 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-befd8f76-ae4e-4071-9d18-f70d38b9b84e None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:25.038030 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-befd8f76-ae4e-4071-9d18-f70d38b9b84e None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:25.042945 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f31fbd-dedb-4f49-aba4-0edd76eb4e6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:25.043551 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f31fbd-dedb-4f49-aba4-0edd76eb4e6b admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:25.043907 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f31fbd-dedb-4f49-aba4-0edd76eb4e6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:25.044178 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f31fbd-dedb-4f49-aba4-0edd76eb4e6b admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:25.044977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f31fbd-dedb-4f49-aba4-0edd76eb4e6b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:10:25.045281 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f31fbd-dedb-4f49-aba4-0edd76eb4e6b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:25.046957 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f31fbd-dedb-4f49-aba4-0edd76eb4e6b admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:25.060262 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1364/2745] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:10:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:10:25.064649 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402dd8d7-808b-475d-990b-9a196ca8c299 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:25.074972 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402dd8d7-808b-475d-990b-9a196ca8c299 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:25.079901 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adddb1a3-0a15-429e-9844-10fc90751bc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:25.080492 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adddb1a3-0a15-429e-9844-10fc90751bc3 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:25.080781 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adddb1a3-0a15-429e-9844-10fc90751bc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:25.081044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adddb1a3-0a15-429e-9844-10fc90751bc3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:25.081704 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adddb1a3-0a15-429e-9844-10fc90751bc3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:25.084467 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adddb1a3-0a15-429e-9844-10fc90751bc3 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:25.182085 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1382/2746] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:10:25 2022] POST /identity/v3/projects => generated 375 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:10:25.186212 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f31fbd-dedb-4f49-aba4-0edd76eb4e6b admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:25.196395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f31fbd-dedb-4f49-aba4-0edd76eb4e6b admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:25.253051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12d612d-b6da-4059-8c6b-06816bde3eb4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:25.253588 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12d612d-b6da-4059-8c6b-06816bde3eb4 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:25.253869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12d612d-b6da-4059-8c6b-06816bde3eb4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:25.254140 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12d612d-b6da-4059-8c6b-06816bde3eb4 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:25.254920 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12d612d-b6da-4059-8c6b-06816bde3eb4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:25.257008 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12d612d-b6da-4059-8c6b-06816bde3eb4 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:25.335922 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1365/2747] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:10:25 2022] POST /identity/v3/users => generated 288 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:10:25.340070 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adddb1a3-0a15-429e-9844-10fc90751bc3 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:25.350924 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adddb1a3-0a15-429e-9844-10fc90751bc3 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:25.355636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82d939b-b5ef-43e2-85e5-3693bfcaf302 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:25.356229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82d939b-b5ef-43e2-85e5-3693bfcaf302 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:25.356507 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82d939b-b5ef-43e2-85e5-3693bfcaf302 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:25.356775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82d939b-b5ef-43e2-85e5-3693bfcaf302 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:25.357415 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82d939b-b5ef-43e2-85e5-3693bfcaf302 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:25.359070 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82d939b-b5ef-43e2-85e5-3693bfcaf302 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:25.369660 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1383/2748] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:10:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:10:25.373697 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12d612d-b6da-4059-8c6b-06816bde3eb4 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:25.383927 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12d612d-b6da-4059-8c6b-06816bde3eb4 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:25.388808 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f985a5e1-a2ba-4125-8248-33b6658cfbaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:25.389431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f985a5e1-a2ba-4125-8248-33b6658cfbaf admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:25.389817 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f985a5e1-a2ba-4125-8248-33b6658cfbaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:25.390086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f985a5e1-a2ba-4125-8248-33b6658cfbaf admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:25.390754 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f985a5e1-a2ba-4125-8248-33b6658cfbaf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:25.392403 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f985a5e1-a2ba-4125-8248-33b6658cfbaf admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:25.405404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1366/2749] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:10:25 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:10:25.409302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82d939b-b5ef-43e2-85e5-3693bfcaf302 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:25.421617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82d939b-b5ef-43e2-85e5-3693bfcaf302 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:25.426484 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9abc60-36be-4226-bc34-b6da159c03c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:25.427191 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9abc60-36be-4226-bc34-b6da159c03c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:25.427495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9abc60-36be-4226-bc34-b6da159c03c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:25.427782 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9abc60-36be-4226-bc34-b6da159c03c7 admin admin] PATH_INFO: `/v3/projects/403c04c19c0c45398d6b232c03a4b7a6/users/1e18b5b70f8b4939bebae2d7ef261740/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:25.443222 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9abc60-36be-4226-bc34-b6da159c03c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=403c04c19c0c45398d6b232c03a4b7a6, user_id=1e18b5b70f8b4939bebae2d7ef261740, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:25.446076 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd9abc60-36be-4226-bc34-b6da159c03c7 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:25.531665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1384/2750] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:10:25 2022] PUT /identity/v3/projects/403c04c19c0c45398d6b232c03a4b7a6/users/1e18b5b70f8b4939bebae2d7ef261740/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:10:25.537839 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1facd3-89e4-403c-9923-ba3227858b69 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:25.538142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1facd3-89e4-403c-9923-ba3227858b69 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:25.538417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1facd3-89e4-403c-9923-ba3227858b69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:25.560598 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-9d1facd3-89e4-403c-9923-ba3227858b69 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:10:25.561802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9d1facd3-89e4-403c-9923-ba3227858b69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:25.622513 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1367/2751] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:10:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:10:25.632339 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9abc60-36be-4226-bc34-b6da159c03c7 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:25.634095 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-fd9abc60-36be-4226-bc34-b6da159c03c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:25.658264 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:10:25.695305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9abc60-36be-4226-bc34-b6da159c03c7 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:25.699079 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b08ac6c-3228-4a7f-ad6b-e6cf264e7821 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:25.699620 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b08ac6c-3228-4a7f-ad6b-e6cf264e7821 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:25.699903 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b08ac6c-3228-4a7f-ad6b-e6cf264e7821 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:25.700177 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b08ac6c-3228-4a7f-ad6b-e6cf264e7821 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:25.710030 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b08ac6c-3228-4a7f-ad6b-e6cf264e7821 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:25.711568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b08ac6c-3228-4a7f-ad6b-e6cf264e7821 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:25.721872 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1385/2752] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:10:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:10:29.739695 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1facd3-89e4-403c-9923-ba3227858b69 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:29.741886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9d1facd3-89e4-403c-9923-ba3227858b69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:29.758160 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1facd3-89e4-403c-9923-ba3227858b69 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:29.763642 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1185ea0-06d3-4c37-ac24-f583df3f400d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:29.764323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1185ea0-06d3-4c37-ac24-f583df3f400d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:29.773769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1185ea0-06d3-4c37-ac24-f583df3f400d admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:29.773769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1185ea0-06d3-4c37-ac24-f583df3f400d admin admin] PATH_INFO: `/v3/users/178e5adbb5634ccdbba91ace9eac23fb` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:29.773769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1185ea0-06d3-4c37-ac24-f583df3f400d admin admin] RBAC: Authorizing `identity:delete_user(user_id=178e5adbb5634ccdbba91ace9eac23fb)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:29.773769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1185ea0-06d3-4c37-ac24-f583df3f400d admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:29.784556 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:10:30.057695 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e1185ea0-06d3-4c37-ac24-f583df3f400d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '178e5adbb5634ccdbba91ace9eac23fb'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:30.188677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e1185ea0-06d3-4c37-ac24-f583df3f400d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '178e5adbb5634ccdbba91ace9eac23fb'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:30.189556 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e1185ea0-06d3-4c37-ac24-f583df3f400d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '178e5adbb5634ccdbba91ace9eac23fb'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:30.217565 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e1185ea0-06d3-4c37-ac24-f583df3f400d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '178e5adbb5634ccdbba91ace9eac23fb'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:30.230916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-e1185ea0-06d3-4c37-ac24-f583df3f400d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '178e5adbb5634ccdbba91ace9eac23fb'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:30.237235 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1368/2753] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:10:29 2022] DELETE /identity/v3/users/178e5adbb5634ccdbba91ace9eac23fb => generated 0 bytes in 498 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:10:30.243606 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7d9313-b70d-4fce-ad0d-cfb49ca9d1f9 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:30.244111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7d9313-b70d-4fce-ad0d-cfb49ca9d1f9 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:30.244379 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7d9313-b70d-4fce-ad0d-cfb49ca9d1f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:30.265849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-fb7d9313-b70d-4fce-ad0d-cfb49ca9d1f9 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:10:30.267267 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-fb7d9313-b70d-4fce-ad0d-cfb49ca9d1f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:30.337715 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1386/2754] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:10:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:10:30.347544 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1185ea0-06d3-4c37-ac24-f583df3f400d admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:30.349857 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e1185ea0-06d3-4c37-ac24-f583df3f400d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:30.452046 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1185ea0-06d3-4c37-ac24-f583df3f400d admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:30.455695 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02b719c-69e6-4988-ab11-90a54ac2356c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:30.456294 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02b719c-69e6-4988-ab11-90a54ac2356c service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:30.456569 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02b719c-69e6-4988-ab11-90a54ac2356c service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:30.456838 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02b719c-69e6-4988-ab11-90a54ac2356c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:30.466751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02b719c-69e6-4988-ab11-90a54ac2356c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:30.468154 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02b719c-69e6-4988-ab11-90a54ac2356c service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:30.479121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1369/2755] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:10:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:10:31.108771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7d9313-b70d-4fce-ad0d-cfb49ca9d1f9 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:31.110298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-fb7d9313-b70d-4fce-ad0d-cfb49ca9d1f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:31.126462 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7d9313-b70d-4fce-ad0d-cfb49ca9d1f9 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:31.131436 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1292aaca-a737-48fc-8600-8d0348be3686 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:31.132193 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1292aaca-a737-48fc-8600-8d0348be3686 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:31.132554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1292aaca-a737-48fc-8600-8d0348be3686 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:31.132922 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1292aaca-a737-48fc-8600-8d0348be3686 admin admin] PATH_INFO: `/v3/projects/c36dff07dcb04ee29210ebac9eb03804` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:31.134166 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1292aaca-a737-48fc-8600-8d0348be3686 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c36dff07dcb04ee29210ebac9eb03804)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:31.136919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1292aaca-a737-48fc-8600-8d0348be3686 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:31.299830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-1292aaca-a737-48fc-8600-8d0348be3686 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c36dff07dcb04ee29210ebac9eb03804', 'request_id': 'req-1292aaca-a737-48fc-8600-8d0348be3686'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:31.379879 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-1292aaca-a737-48fc-8600-8d0348be3686 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c36dff07dcb04ee29210ebac9eb03804', 'request_id': 'req-1292aaca-a737-48fc-8600-8d0348be3686'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:31.390467 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-1292aaca-a737-48fc-8600-8d0348be3686 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c36dff07dcb04ee29210ebac9eb03804', 'request_id': 'req-1292aaca-a737-48fc-8600-8d0348be3686'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:31.440867 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-1292aaca-a737-48fc-8600-8d0348be3686 admin admin] The token cache is being invalidate because project c36dff07dcb04ee29210ebac9eb03804 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:10:31.441373 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-1292aaca-a737-48fc-8600-8d0348be3686 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:31.459257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-1292aaca-a737-48fc-8600-8d0348be3686 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:10:31.459257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1387/2756] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:10:31 2022] DELETE /identity/v3/projects/c36dff07dcb04ee29210ebac9eb03804 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:10:31.741494 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d81558c-bdb3-4193-a0d3-20c4d6b2e029 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:31.742069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d81558c-bdb3-4193-a0d3-20c4d6b2e029 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:31.742069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d81558c-bdb3-4193-a0d3-20c4d6b2e029 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:31.777441 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-2d81558c-bdb3-4193-a0d3-20c4d6b2e029 None None] MFA Rules not processed for user `1e18b5b70f8b4939bebae2d7ef261740`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:10:31.778718 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-2d81558c-bdb3-4193-a0d3-20c4d6b2e029 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:31.855956 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1370/2757] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:10:31 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:10:31.870776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1292aaca-a737-48fc-8600-8d0348be3686 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:31.872591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-1292aaca-a737-48fc-8600-8d0348be3686 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:31.941399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:10:31.996633 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1292aaca-a737-48fc-8600-8d0348be3686 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:32.002456 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303f308b-431a-4d42-a652-111d189f88b6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:32.003077 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303f308b-431a-4d42-a652-111d189f88b6 service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:32.003353 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303f308b-431a-4d42-a652-111d189f88b6 service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:32.003622 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303f308b-431a-4d42-a652-111d189f88b6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:32.014824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303f308b-431a-4d42-a652-111d189f88b6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:32.016832 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303f308b-431a-4d42-a652-111d189f88b6 service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:32.029903 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1388/2758] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:10:31 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:10:35.657449 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e65fb7-adb4-4910-87e4-9d15afa2f81d None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:35.657950 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e65fb7-adb4-4910-87e4-9d15afa2f81d None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:35.658108 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e65fb7-adb4-4910-87e4-9d15afa2f81d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:35.665323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:10:35.677925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-a5e65fb7-adb4-4910-87e4-9d15afa2f81d None None] MFA Rules not processed for user `1e18b5b70f8b4939bebae2d7ef261740`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:10:35.679261 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a5e65fb7-adb4-4910-87e4-9d15afa2f81d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:35.686939 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1371/2759] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:10:35 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:10:38.747262 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303f308b-431a-4d42-a652-111d189f88b6 service nova] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:38.748672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-303f308b-431a-4d42-a652-111d189f88b6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:38.841757 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303f308b-431a-4d42-a652-111d189f88b6 service nova] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:38.846792 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a812b66-d92b-4a57-a6e5-b1baa722f3c2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce249d4a99c34891b6ba941f57a14dd4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:38.847385 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a812b66-d92b-4a57-a6e5-b1baa722f3c2 service placement] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:38.847658 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a812b66-d92b-4a57-a6e5-b1baa722f3c2 service placement] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:38.847921 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a812b66-d92b-4a57-a6e5-b1baa722f3c2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:38.849651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-5a812b66-d92b-4a57-a6e5-b1baa722f3c2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:38.860771 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a812b66-d92b-4a57-a6e5-b1baa722f3c2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:38.862362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a812b66-d92b-4a57-a6e5-b1baa722f3c2 service placement] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:38.872628 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1389/2760] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:10:38 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:10:46.042335 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:10:46.077788 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e65fb7-adb4-4910-87e4-9d15afa2f81d None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:46.079302 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-a5e65fb7-adb4-4910-87e4-9d15afa2f81d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:46.135624 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e65fb7-adb4-4910-87e4-9d15afa2f81d None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:46.139230 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da217745-8627-40a6-9bfc-0731f0b8dfc6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:46.139946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da217745-8627-40a6-9bfc-0731f0b8dfc6 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:46.140226 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da217745-8627-40a6-9bfc-0731f0b8dfc6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:46.140500 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da217745-8627-40a6-9bfc-0731f0b8dfc6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:46.142341 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-da217745-8627-40a6-9bfc-0731f0b8dfc6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:46.202433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da217745-8627-40a6-9bfc-0731f0b8dfc6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:46.203868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da217745-8627-40a6-9bfc-0731f0b8dfc6 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:46.214189 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1372/2761] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:10:46 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:10:48.868433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=66 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:10:51.999041 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b38685-43fe-4d4c-a550-58c6613a8af2 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:51.999600 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b38685-43fe-4d4c-a550-58c6613a8af2 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:51.999600 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b38685-43fe-4d4c-a550-58c6613a8af2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:52.030917 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-86b38685-43fe-4d4c-a550-58c6613a8af2 None None] MFA Rules not processed for user `b471d768cd814ff8b4f6bd35d21e24a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:10:52.032127 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-86b38685-43fe-4d4c-a550-58c6613a8af2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:52.089448 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1390/2762] 10.222.0.62 () {58 vars in 1009 bytes} [Fri Sep 23 14:10:51 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:10:52.280057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da217745-8627-40a6-9bfc-0731f0b8dfc6 service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:52.281473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-da217745-8627-40a6-9bfc-0731f0b8dfc6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:52.353518 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da217745-8627-40a6-9bfc-0731f0b8dfc6 service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:52.358830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb32d77-6e51-48ad-b615-ff5374ec3d1d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b9269509f1d454eab7a3008c2ba0039', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:52.359545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb32d77-6e51-48ad-b615-ff5374ec3d1d service swift] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:52.359888 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb32d77-6e51-48ad-b615-ff5374ec3d1d service swift] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:52.360319 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb32d77-6e51-48ad-b615-ff5374ec3d1d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:52.369543 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb32d77-6e51-48ad-b615-ff5374ec3d1d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:52.371175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb32d77-6e51-48ad-b615-ff5374ec3d1d service swift] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:52.379617 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1373/2763] 10.222.0.62 () {60 vars in 1384 bytes} [Fri Sep 23 14:10:52 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 23 14:10:56.658362 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b38685-43fe-4d4c-a550-58c6613a8af2 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:10:56.671195 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b38685-43fe-4d4c-a550-58c6613a8af2 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:10:56.675447 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b524b94-0c90-419d-b5e0-8c0fc23c9463 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:10:56.676161 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b524b94-0c90-419d-b5e0-8c0fc23c9463 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:10:56.676707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b524b94-0c90-419d-b5e0-8c0fc23c9463 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:10:56.677235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b524b94-0c90-419d-b5e0-8c0fc23c9463 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:10:56.679205 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-7b524b94-0c90-419d-b5e0-8c0fc23c9463 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:10:56.812475 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b524b94-0c90-419d-b5e0-8c0fc23c9463 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:10:56.814168 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b524b94-0c90-419d-b5e0-8c0fc23c9463 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:10:56.831764 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1391/2764] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:10:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:11:02.376559 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=80 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:11:06.167225 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb32d77-6e51-48ad-b615-ff5374ec3d1d service swift] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:06.181037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-0eb32d77-6e51-48ad-b615-ff5374ec3d1d service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:11:06.212830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb32d77-6e51-48ad-b615-ff5374ec3d1d service swift] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:06.221068 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225073a6-f562-4add-9f28-e34936f06cae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:06.221863 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225073a6-f562-4add-9f28-e34936f06cae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:06.222243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225073a6-f562-4add-9f28-e34936f06cae admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:06.222648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225073a6-f562-4add-9f28-e34936f06cae admin admin] PATH_INFO: `/v3/users/4ad2b6e8ec52485488f27ea84f2c69cc` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:06.223898 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225073a6-f562-4add-9f28-e34936f06cae admin admin] RBAC: Authorizing `identity:delete_user(user_id=4ad2b6e8ec52485488f27ea84f2c69cc)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:06.225950 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225073a6-f562-4add-9f28-e34936f06cae admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:06.416574 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-225073a6-f562-4add-9f28-e34936f06cae admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4ad2b6e8ec52485488f27ea84f2c69cc'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:06.490452 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-225073a6-f562-4add-9f28-e34936f06cae admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4ad2b6e8ec52485488f27ea84f2c69cc'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:06.491162 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-225073a6-f562-4add-9f28-e34936f06cae admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4ad2b6e8ec52485488f27ea84f2c69cc'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:06.514655 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-225073a6-f562-4add-9f28-e34936f06cae admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4ad2b6e8ec52485488f27ea84f2c69cc'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:06.524546 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-225073a6-f562-4add-9f28-e34936f06cae admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4ad2b6e8ec52485488f27ea84f2c69cc'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:06.534423 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1374/2765] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:11:06 2022] DELETE /identity/v3/users/4ad2b6e8ec52485488f27ea84f2c69cc => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:11:06.824231 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=59 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:11:06.843868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b524b94-0c90-419d-b5e0-8c0fc23c9463 service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:06.853860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b524b94-0c90-419d-b5e0-8c0fc23c9463 service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:06.957867 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78747199-114d-45a1-8672-58e49382ca09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:06.958672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78747199-114d-45a1-8672-58e49382ca09 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:06.958978 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78747199-114d-45a1-8672-58e49382ca09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:06.959321 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78747199-114d-45a1-8672-58e49382ca09 admin admin] PATH_INFO: `/v3/projects/728460a287914347b74cfe886f56ee6c` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:06.960538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78747199-114d-45a1-8672-58e49382ca09 admin admin] RBAC: Authorizing `identity:delete_project(project_id=728460a287914347b74cfe886f56ee6c)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:06.962582 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78747199-114d-45a1-8672-58e49382ca09 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:07.114429 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-78747199-114d-45a1-8672-58e49382ca09 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '728460a287914347b74cfe886f56ee6c', 'request_id': 'req-78747199-114d-45a1-8672-58e49382ca09'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:07.184705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-78747199-114d-45a1-8672-58e49382ca09 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '728460a287914347b74cfe886f56ee6c', 'request_id': 'req-78747199-114d-45a1-8672-58e49382ca09'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:07.193435 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-78747199-114d-45a1-8672-58e49382ca09 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '728460a287914347b74cfe886f56ee6c', 'request_id': 'req-78747199-114d-45a1-8672-58e49382ca09'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:07.206623 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-78747199-114d-45a1-8672-58e49382ca09 admin admin] The token cache is being invalidate because project 728460a287914347b74cfe886f56ee6c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:11:07.214299 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-78747199-114d-45a1-8672-58e49382ca09 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:07.214299 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-78747199-114d-45a1-8672-58e49382ca09 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:07.214299 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1392/2766] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:11:06 2022] DELETE /identity/v3/projects/728460a287914347b74cfe886f56ee6c => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:11:07.215186 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225073a6-f562-4add-9f28-e34936f06cae admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:07.217689 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-225073a6-f562-4add-9f28-e34936f06cae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:11:07.236660 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225073a6-f562-4add-9f28-e34936f06cae admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:07.314810 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d345f4-f56d-4e8d-8fa4-73d777f140fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:07.315887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d345f4-f56d-4e8d-8fa4-73d777f140fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:07.316333 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d345f4-f56d-4e8d-8fa4-73d777f140fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:07.317044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d345f4-f56d-4e8d-8fa4-73d777f140fd admin admin] PATH_INFO: `/v3/users/4fa66c39478e46ef9c69e3c1ad446252` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:07.318794 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d345f4-f56d-4e8d-8fa4-73d777f140fd admin admin] RBAC: Authorizing `identity:delete_user(user_id=4fa66c39478e46ef9c69e3c1ad446252)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:07.321390 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d345f4-f56d-4e8d-8fa4-73d777f140fd admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:07.514663 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-22d345f4-f56d-4e8d-8fa4-73d777f140fd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4fa66c39478e46ef9c69e3c1ad446252'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:07.621253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-22d345f4-f56d-4e8d-8fa4-73d777f140fd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4fa66c39478e46ef9c69e3c1ad446252'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:07.622062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-22d345f4-f56d-4e8d-8fa4-73d777f140fd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4fa66c39478e46ef9c69e3c1ad446252'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:07.637091 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-22d345f4-f56d-4e8d-8fa4-73d777f140fd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4fa66c39478e46ef9c69e3c1ad446252'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:07.639703 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:11:07.645992 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-22d345f4-f56d-4e8d-8fa4-73d777f140fd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4fa66c39478e46ef9c69e3c1ad446252'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:07.652186 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1375/2767] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:11:07 2022] DELETE /identity/v3/users/4fa66c39478e46ef9c69e3c1ad446252 => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:11:07.941209 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78747199-114d-45a1-8672-58e49382ca09 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:07.943916 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-78747199-114d-45a1-8672-58e49382ca09 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:11:07.977186 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78747199-114d-45a1-8672-58e49382ca09 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:08.065155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df426d5d-fa01-46b0-974d-7a4526ee1024 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:08.066111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df426d5d-fa01-46b0-974d-7a4526ee1024 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:08.067730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df426d5d-fa01-46b0-974d-7a4526ee1024 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:08.068243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df426d5d-fa01-46b0-974d-7a4526ee1024 admin admin] PATH_INFO: `/v3/projects/09f3cfa0c4164cb0b07f49311387aa98` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:08.069998 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df426d5d-fa01-46b0-974d-7a4526ee1024 admin admin] RBAC: Authorizing `identity:delete_project(project_id=09f3cfa0c4164cb0b07f49311387aa98)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:08.072658 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df426d5d-fa01-46b0-974d-7a4526ee1024 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:08.189415 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:11:08.209024 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-df426d5d-fa01-46b0-974d-7a4526ee1024 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '09f3cfa0c4164cb0b07f49311387aa98', 'request_id': 'req-df426d5d-fa01-46b0-974d-7a4526ee1024'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:08.261804 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-df426d5d-fa01-46b0-974d-7a4526ee1024 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '09f3cfa0c4164cb0b07f49311387aa98', 'request_id': 'req-df426d5d-fa01-46b0-974d-7a4526ee1024'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:08.268834 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-df426d5d-fa01-46b0-974d-7a4526ee1024 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '09f3cfa0c4164cb0b07f49311387aa98', 'request_id': 'req-df426d5d-fa01-46b0-974d-7a4526ee1024'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:08.281188 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-df426d5d-fa01-46b0-974d-7a4526ee1024 admin admin] The token cache is being invalidate because project 09f3cfa0c4164cb0b07f49311387aa98 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:11:08.281646 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-df426d5d-fa01-46b0-974d-7a4526ee1024 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:08.283618 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-df426d5d-fa01-46b0-974d-7a4526ee1024 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:08.284412 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1393/2768] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:11:07 2022] DELETE /identity/v3/projects/09f3cfa0c4164cb0b07f49311387aa98 => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:11:11.562223 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89807662-3d9b-43d6-a275-e6f5c7bacdde None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:11.562682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89807662-3d9b-43d6-a275-e6f5c7bacdde None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:11.562787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89807662-3d9b-43d6-a275-e6f5c7bacdde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:11.581600 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-89807662-3d9b-43d6-a275-e6f5c7bacdde None None] MFA Rules not processed for user `b471d768cd814ff8b4f6bd35d21e24a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:11:11.582830 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-89807662-3d9b-43d6-a275-e6f5c7bacdde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:11:11.636650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1376/2769] 10.222.0.62 () {58 vars in 967 bytes} [Fri Sep 23 14:11:11 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:11:11.648997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df426d5d-fa01-46b0-974d-7a4526ee1024 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:11.650454 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-df426d5d-fa01-46b0-974d-7a4526ee1024 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:11:11.719159 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df426d5d-fa01-46b0-974d-7a4526ee1024 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:11.724616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca8aced-9f8f-4774-801c-09bd18dbeb53 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b9269509f1d454eab7a3008c2ba0039', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:11.725319 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca8aced-9f8f-4774-801c-09bd18dbeb53 service swift] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:11.725602 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca8aced-9f8f-4774-801c-09bd18dbeb53 service swift] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:11.725876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca8aced-9f8f-4774-801c-09bd18dbeb53 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:11.736350 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca8aced-9f8f-4774-801c-09bd18dbeb53 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:11.737917 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca8aced-9f8f-4774-801c-09bd18dbeb53 service swift] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:11.746668 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1394/2770] 10.222.0.62 () {60 vars in 1384 bytes} [Fri Sep 23 14:11:11 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 23 14:11:14.805022 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89807662-3d9b-43d6-a275-e6f5c7bacdde None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:14.807284 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-89807662-3d9b-43d6-a275-e6f5c7bacdde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:11:14.875405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89807662-3d9b-43d6-a275-e6f5c7bacdde None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:14.880489 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a592880-920c-43c2-8fff-6c78002f09b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:14.881112 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a592880-920c-43c2-8fff-6c78002f09b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:14.881389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a592880-920c-43c2-8fff-6c78002f09b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:14.881658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a592880-920c-43c2-8fff-6c78002f09b4 admin admin] PATH_INFO: `/v3/users/256d2963df4c4e48876583485de59674` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:14.882807 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a592880-920c-43c2-8fff-6c78002f09b4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=256d2963df4c4e48876583485de59674)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:14.884705 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a592880-920c-43c2-8fff-6c78002f09b4 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:15.067797 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-8a592880-920c-43c2-8fff-6c78002f09b4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '256d2963df4c4e48876583485de59674'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:15.144164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-8a592880-920c-43c2-8fff-6c78002f09b4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '256d2963df4c4e48876583485de59674'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:15.145277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-8a592880-920c-43c2-8fff-6c78002f09b4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '256d2963df4c4e48876583485de59674'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:15.158323 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-8a592880-920c-43c2-8fff-6c78002f09b4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '256d2963df4c4e48876583485de59674'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:15.165948 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-8a592880-920c-43c2-8fff-6c78002f09b4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '256d2963df4c4e48876583485de59674'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:15.171940 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1377/2771] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:11:14 2022] DELETE /identity/v3/users/256d2963df4c4e48876583485de59674 => generated 0 bytes in 367 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:11:15.419481 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca8aced-9f8f-4774-801c-09bd18dbeb53 service swift] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:15.431929 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca8aced-9f8f-4774-801c-09bd18dbeb53 service swift] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:15.489169 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830fe80a-397f-4af0-9531-2ada23df397a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:15.489790 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830fe80a-397f-4af0-9531-2ada23df397a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:15.490064 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830fe80a-397f-4af0-9531-2ada23df397a admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:15.490412 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830fe80a-397f-4af0-9531-2ada23df397a admin admin] PATH_INFO: `/v3/projects/c4ce494070ca4b76a93a6a4a9f42414c` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:15.491471 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830fe80a-397f-4af0-9531-2ada23df397a admin admin] RBAC: Authorizing `identity:delete_project(project_id=c4ce494070ca4b76a93a6a4a9f42414c)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:15.493348 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830fe80a-397f-4af0-9531-2ada23df397a admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:15.580454 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-830fe80a-397f-4af0-9531-2ada23df397a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c4ce494070ca4b76a93a6a4a9f42414c', 'request_id': 'req-830fe80a-397f-4af0-9531-2ada23df397a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:15.583651 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=9,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:11:15.678129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-830fe80a-397f-4af0-9531-2ada23df397a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c4ce494070ca4b76a93a6a4a9f42414c', 'request_id': 'req-830fe80a-397f-4af0-9531-2ada23df397a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:15.684314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-830fe80a-397f-4af0-9531-2ada23df397a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c4ce494070ca4b76a93a6a4a9f42414c', 'request_id': 'req-830fe80a-397f-4af0-9531-2ada23df397a'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:15.697082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-830fe80a-397f-4af0-9531-2ada23df397a admin admin] The token cache is being invalidate because project c4ce494070ca4b76a93a6a4a9f42414c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:11:15.697374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-830fe80a-397f-4af0-9531-2ada23df397a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:15.698009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-830fe80a-397f-4af0-9531-2ada23df397a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:15.699575 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1395/2772] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:11:15 2022] DELETE /identity/v3/projects/c4ce494070ca4b76a93a6a4a9f42414c => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:11:15.704521 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6facf6-7af1-425f-993a-a8fa52fc1e23 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:15.704784 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6facf6-7af1-425f-993a-a8fa52fc1e23 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:15.705079 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6facf6-7af1-425f-993a-a8fa52fc1e23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:15.724781 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-ce6facf6-7af1-425f-993a-a8fa52fc1e23 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:11:15.725981 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ce6facf6-7af1-425f-993a-a8fa52fc1e23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:11:15.735460 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:11:15.784504 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1378/2773] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:11:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:11:15.792407 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e2e7b4-edff-4ece-8472-8e6388dfcc5f None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:15.792718 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e2e7b4-edff-4ece-8472-8e6388dfcc5f None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:15.793014 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e2e7b4-edff-4ece-8472-8e6388dfcc5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:15.814226 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-42e2e7b4-edff-4ece-8472-8e6388dfcc5f None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:11:15.815466 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-42e2e7b4-edff-4ece-8472-8e6388dfcc5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:11:15.824759 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1396/2774] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:11:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:11:15.829042 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6facf6-7af1-425f-993a-a8fa52fc1e23 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:15.839374 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6facf6-7af1-425f-993a-a8fa52fc1e23 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:15.843995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac98d1e-265a-4fa1-a481-e230895a13f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:15.844587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac98d1e-265a-4fa1-a481-e230895a13f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:15.844864 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac98d1e-265a-4fa1-a481-e230895a13f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:15.845132 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eac98d1e-265a-4fa1-a481-e230895a13f0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:15.845769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac98d1e-265a-4fa1-a481-e230895a13f0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:11:15.846068 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac98d1e-265a-4fa1-a481-e230895a13f0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:15.848275 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eac98d1e-265a-4fa1-a481-e230895a13f0 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:15.859696 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1379/2775] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:11:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:11:15.863998 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e2e7b4-edff-4ece-8472-8e6388dfcc5f None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:15.873152 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e2e7b4-edff-4ece-8472-8e6388dfcc5f None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:15.877827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a6d53a-c0b4-4dc8-92d1-5d5209b99e16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:15.878445 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a6d53a-c0b4-4dc8-92d1-5d5209b99e16 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:15.878723 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a6d53a-c0b4-4dc8-92d1-5d5209b99e16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:15.878995 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a6d53a-c0b4-4dc8-92d1-5d5209b99e16 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:15.879672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a6d53a-c0b4-4dc8-92d1-5d5209b99e16 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:15.881542 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a6d53a-c0b4-4dc8-92d1-5d5209b99e16 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:15.996922 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1397/2776] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:11:15 2022] POST /identity/v3/projects => generated 399 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:11:16.001142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac98d1e-265a-4fa1-a481-e230895a13f0 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:16.011876 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eac98d1e-265a-4fa1-a481-e230895a13f0 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:16.068857 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938b8e32-107d-4c85-a336-5241d24ed366 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:16.069453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938b8e32-107d-4c85-a336-5241d24ed366 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:16.069771 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938b8e32-107d-4c85-a336-5241d24ed366 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:16.070039 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938b8e32-107d-4c85-a336-5241d24ed366 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:16.070694 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938b8e32-107d-4c85-a336-5241d24ed366 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:16.072581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938b8e32-107d-4c85-a336-5241d24ed366 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:16.170067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1380/2777] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:11:16 2022] POST /identity/v3/users => generated 300 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:11:16.174160 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a6d53a-c0b4-4dc8-92d1-5d5209b99e16 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:16.185347 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a6d53a-c0b4-4dc8-92d1-5d5209b99e16 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:16.190090 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c3348a-58bf-4a5a-9af9-27a6d10cb475 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:16.190707 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c3348a-58bf-4a5a-9af9-27a6d10cb475 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:16.191072 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c3348a-58bf-4a5a-9af9-27a6d10cb475 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:16.191971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c3348a-58bf-4a5a-9af9-27a6d10cb475 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:16.192669 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c3348a-58bf-4a5a-9af9-27a6d10cb475 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:16.194513 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c3348a-58bf-4a5a-9af9-27a6d10cb475 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:16.204742 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1398/2778] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:11:16 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:11:16.209410 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938b8e32-107d-4c85-a336-5241d24ed366 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:16.220401 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938b8e32-107d-4c85-a336-5241d24ed366 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:16.224804 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7682d9a-0839-4afd-9d0a-a8aa132fb4aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:16.225402 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7682d9a-0839-4afd-9d0a-a8aa132fb4aa admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:16.225682 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7682d9a-0839-4afd-9d0a-a8aa132fb4aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:16.225952 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7682d9a-0839-4afd-9d0a-a8aa132fb4aa admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:16.226627 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7682d9a-0839-4afd-9d0a-a8aa132fb4aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:16.228200 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7682d9a-0839-4afd-9d0a-a8aa132fb4aa admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:16.238837 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1381/2779] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:11:16 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:11:16.243102 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c3348a-58bf-4a5a-9af9-27a6d10cb475 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:16.253237 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c3348a-58bf-4a5a-9af9-27a6d10cb475 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:16.258356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17dcb51-4651-4c42-8f34-e63ae02cac58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:16.259007 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17dcb51-4651-4c42-8f34-e63ae02cac58 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:16.259286 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17dcb51-4651-4c42-8f34-e63ae02cac58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:16.259555 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17dcb51-4651-4c42-8f34-e63ae02cac58 admin admin] PATH_INFO: `/v3/projects/7bb33ccaa0a148899e9929db72020fd7/users/e11f119fafab401ea09223e5083c01d1/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:16.274630 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17dcb51-4651-4c42-8f34-e63ae02cac58 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7bb33ccaa0a148899e9929db72020fd7, user_id=e11f119fafab401ea09223e5083c01d1, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:16.277266 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17dcb51-4651-4c42-8f34-e63ae02cac58 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:16.360617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1399/2780] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:11:16 2022] PUT /identity/v3/projects/7bb33ccaa0a148899e9929db72020fd7/users/e11f119fafab401ea09223e5083c01d1/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:11:16.371391 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d751039-9a15-4f6b-9c5c-ad9022f07abb None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:16.371673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d751039-9a15-4f6b-9c5c-ad9022f07abb None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:16.371947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d751039-9a15-4f6b-9c5c-ad9022f07abb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:16.400235 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-7d751039-9a15-4f6b-9c5c-ad9022f07abb None None] MFA Rules not processed for user `e11f119fafab401ea09223e5083c01d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:11:16.401445 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-7d751039-9a15-4f6b-9c5c-ad9022f07abb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:11:16.464997 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1382/2781] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:11:16 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:11:16.477140 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17dcb51-4651-4c42-8f34-e63ae02cac58 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:16.478867 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-b17dcb51-4651-4c42-8f34-e63ae02cac58 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:11:16.556165 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17dcb51-4651-4c42-8f34-e63ae02cac58 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:16.559732 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1d5365-e180-4c70-ab7b-970940a35675 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:16.560321 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1d5365-e180-4c70-ab7b-970940a35675 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:16.560593 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1d5365-e180-4c70-ab7b-970940a35675 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:16.560859 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1d5365-e180-4c70-ab7b-970940a35675 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:16.570008 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1d5365-e180-4c70-ab7b-970940a35675 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:16.571491 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1d5365-e180-4c70-ab7b-970940a35675 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:16.580239 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1400/2782] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:11:16 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:11:25.579366 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d751039-9a15-4f6b-9c5c-ad9022f07abb None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:25.597987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:11:25.670701 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d751039-9a15-4f6b-9c5c-ad9022f07abb None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:25.675749 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9176e724-1917-4721-af14-5814f204490a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:25.676603 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9176e724-1917-4721-af14-5814f204490a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:25.676886 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9176e724-1917-4721-af14-5814f204490a admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:25.677718 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9176e724-1917-4721-af14-5814f204490a admin admin] PATH_INFO: `/v3/users/e11f119fafab401ea09223e5083c01d1` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:25.679041 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9176e724-1917-4721-af14-5814f204490a admin admin] RBAC: Authorizing `identity:delete_user(user_id=e11f119fafab401ea09223e5083c01d1)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:25.682252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9176e724-1917-4721-af14-5814f204490a admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:25.840950 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-9176e724-1917-4721-af14-5814f204490a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e11f119fafab401ea09223e5083c01d1'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:25.964389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-9176e724-1917-4721-af14-5814f204490a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e11f119fafab401ea09223e5083c01d1'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:25.965048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-9176e724-1917-4721-af14-5814f204490a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e11f119fafab401ea09223e5083c01d1'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:25.979419 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-9176e724-1917-4721-af14-5814f204490a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e11f119fafab401ea09223e5083c01d1'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:25.986678 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-9176e724-1917-4721-af14-5814f204490a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e11f119fafab401ea09223e5083c01d1'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:25.993624 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1383/2783] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:11:25 2022] DELETE /identity/v3/users/e11f119fafab401ea09223e5083c01d1 => generated 0 bytes in 415 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:11:25.998206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd20d248-bdba-495d-9a06-04ed5a314a14 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:25.998722 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd20d248-bdba-495d-9a06-04ed5a314a14 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:25.998828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd20d248-bdba-495d-9a06-04ed5a314a14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:26.018364 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-dd20d248-bdba-495d-9a06-04ed5a314a14 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:11:26.019670 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-dd20d248-bdba-495d-9a06-04ed5a314a14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:11:26.077787 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:DELETE=3,keystone:SELECT=95,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:11:26.091751 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1401/2784] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:11:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:11:26.102142 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9176e724-1917-4721-af14-5814f204490a admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:26.103813 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-9176e724-1917-4721-af14-5814f204490a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:11:26.165664 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9176e724-1917-4721-af14-5814f204490a admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:26.169468 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5355e78-d4cf-45a8-af5c-30f16f9a235b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:26.172194 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5355e78-d4cf-45a8-af5c-30f16f9a235b service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:26.172194 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5355e78-d4cf-45a8-af5c-30f16f9a235b service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:26.172194 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5355e78-d4cf-45a8-af5c-30f16f9a235b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:26.181114 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5355e78-d4cf-45a8-af5c-30f16f9a235b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:26.182760 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5355e78-d4cf-45a8-af5c-30f16f9a235b service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:26.194249 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1384/2785] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:11:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:11:26.624133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd20d248-bdba-495d-9a06-04ed5a314a14 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:26.625721 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-dd20d248-bdba-495d-9a06-04ed5a314a14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:11:26.638624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd20d248-bdba-495d-9a06-04ed5a314a14 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:26.643436 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c93e565-341e-4eb6-a64b-28bc3638d826 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:26.644083 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c93e565-341e-4eb6-a64b-28bc3638d826 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:26.644332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c93e565-341e-4eb6-a64b-28bc3638d826 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:26.644598 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c93e565-341e-4eb6-a64b-28bc3638d826 admin admin] PATH_INFO: `/v3/projects/7bb33ccaa0a148899e9929db72020fd7` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:26.645878 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c93e565-341e-4eb6-a64b-28bc3638d826 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7bb33ccaa0a148899e9929db72020fd7)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:26.650712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c93e565-341e-4eb6-a64b-28bc3638d826 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:26.752287 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7c93e565-341e-4eb6-a64b-28bc3638d826 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7bb33ccaa0a148899e9929db72020fd7', 'request_id': 'req-7c93e565-341e-4eb6-a64b-28bc3638d826'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:26.855852 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7c93e565-341e-4eb6-a64b-28bc3638d826 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7bb33ccaa0a148899e9929db72020fd7', 'request_id': 'req-7c93e565-341e-4eb6-a64b-28bc3638d826'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:26.864424 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7c93e565-341e-4eb6-a64b-28bc3638d826 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7bb33ccaa0a148899e9929db72020fd7', 'request_id': 'req-7c93e565-341e-4eb6-a64b-28bc3638d826'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:26.878136 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7c93e565-341e-4eb6-a64b-28bc3638d826 admin admin] The token cache is being invalidate because project 7bb33ccaa0a148899e9929db72020fd7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:11:26.884665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7c93e565-341e-4eb6-a64b-28bc3638d826 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:26.884665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-7c93e565-341e-4eb6-a64b-28bc3638d826 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:26.884665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1402/2786] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:11:26 2022] DELETE /identity/v3/projects/7bb33ccaa0a148899e9929db72020fd7 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:11:36.188380 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=74,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:11:36.874924 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=27,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:11:40.794520 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5355e78-d4cf-45a8-af5c-30f16f9a235b service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:40.803555 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5355e78-d4cf-45a8-af5c-30f16f9a235b service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:40.851907 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d534ef-1b6e-4b1a-ad7b-3abaf501fec6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:40.853069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d534ef-1b6e-4b1a-ad7b-3abaf501fec6 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:40.853349 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d534ef-1b6e-4b1a-ad7b-3abaf501fec6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:40.853798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d534ef-1b6e-4b1a-ad7b-3abaf501fec6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:40.855546 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-b0d534ef-1b6e-4b1a-ad7b-3abaf501fec6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:11:40.917551 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d534ef-1b6e-4b1a-ad7b-3abaf501fec6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:40.919021 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d534ef-1b6e-4b1a-ad7b-3abaf501fec6 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:40.928607 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1385/2787] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:11:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:11:48.241356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c93e565-341e-4eb6-a64b-28bc3638d826 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:48.242838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-7c93e565-341e-4eb6-a64b-28bc3638d826 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:11:48.256155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c93e565-341e-4eb6-a64b-28bc3638d826 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:48.262182 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d847df-b137-434e-8a0f-7e8e9685dd75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:48.262839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d847df-b137-434e-8a0f-7e8e9685dd75 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:48.263118 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d847df-b137-434e-8a0f-7e8e9685dd75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:48.263453 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d847df-b137-434e-8a0f-7e8e9685dd75 admin admin] PATH_INFO: `/v3/users/9359e90c4861470d9cc060abb9c3b203` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:48.264678 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d847df-b137-434e-8a0f-7e8e9685dd75 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9359e90c4861470d9cc060abb9c3b203)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:48.266667 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d847df-b137-434e-8a0f-7e8e9685dd75 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:48.440050 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-d3d847df-b137-434e-8a0f-7e8e9685dd75 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9359e90c4861470d9cc060abb9c3b203'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:48.512098 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-d3d847df-b137-434e-8a0f-7e8e9685dd75 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9359e90c4861470d9cc060abb9c3b203'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:48.524815 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-d3d847df-b137-434e-8a0f-7e8e9685dd75 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9359e90c4861470d9cc060abb9c3b203'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:48.527898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-d3d847df-b137-434e-8a0f-7e8e9685dd75 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9359e90c4861470d9cc060abb9c3b203'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:48.533082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-d3d847df-b137-434e-8a0f-7e8e9685dd75 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9359e90c4861470d9cc060abb9c3b203'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:48.539233 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1403/2788] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:11:48 2022] DELETE /identity/v3/users/9359e90c4861470d9cc060abb9c3b203 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:11:48.770967 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d534ef-1b6e-4b1a-ad7b-3abaf501fec6 service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:48.781542 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d534ef-1b6e-4b1a-ad7b-3abaf501fec6 service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:48.847383 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c52cbf1-8597-4cb6-906f-0a134453717c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:48.848007 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c52cbf1-8597-4cb6-906f-0a134453717c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:48.848281 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c52cbf1-8597-4cb6-906f-0a134453717c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:48.848544 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c52cbf1-8597-4cb6-906f-0a134453717c admin admin] PATH_INFO: `/v3/projects/901178273d85449e9cdaa5de235fb3e9` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:48.849650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c52cbf1-8597-4cb6-906f-0a134453717c admin admin] RBAC: Authorizing `identity:delete_project(project_id=901178273d85449e9cdaa5de235fb3e9)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:48.851725 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c52cbf1-8597-4cb6-906f-0a134453717c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:48.959006 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3c52cbf1-8597-4cb6-906f-0a134453717c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '901178273d85449e9cdaa5de235fb3e9', 'request_id': 'req-3c52cbf1-8597-4cb6-906f-0a134453717c'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:48.961446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:11:49.060550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3c52cbf1-8597-4cb6-906f-0a134453717c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '901178273d85449e9cdaa5de235fb3e9', 'request_id': 'req-3c52cbf1-8597-4cb6-906f-0a134453717c'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:49.066475 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3c52cbf1-8597-4cb6-906f-0a134453717c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '901178273d85449e9cdaa5de235fb3e9', 'request_id': 'req-3c52cbf1-8597-4cb6-906f-0a134453717c'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:49.078057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3c52cbf1-8597-4cb6-906f-0a134453717c admin admin] The token cache is being invalidate because project 901178273d85449e9cdaa5de235fb3e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:11:49.078418 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3c52cbf1-8597-4cb6-906f-0a134453717c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:49.079044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3c52cbf1-8597-4cb6-906f-0a134453717c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:49.080538 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1386/2789] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:11:48 2022] DELETE /identity/v3/projects/901178273d85449e9cdaa5de235fb3e9 => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:11:49.084725 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d847df-b137-434e-8a0f-7e8e9685dd75 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:49.086120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d3d847df-b137-434e-8a0f-7e8e9685dd75 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:11:49.099444 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d847df-b137-434e-8a0f-7e8e9685dd75 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:49.156132 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ee7315-ac71-436b-a561-26d4b63f8c82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:49.156748 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ee7315-ac71-436b-a561-26d4b63f8c82 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:49.157023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ee7315-ac71-436b-a561-26d4b63f8c82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:49.157291 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ee7315-ac71-436b-a561-26d4b63f8c82 admin admin] PATH_INFO: `/v3/users/de28bb95bc464874a7f70b33e5c5f8e3` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:49.158425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ee7315-ac71-436b-a561-26d4b63f8c82 admin admin] RBAC: Authorizing `identity:delete_user(user_id=de28bb95bc464874a7f70b33e5c5f8e3)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:49.160336 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ee7315-ac71-436b-a561-26d4b63f8c82 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:49.337152 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-05ee7315-ac71-436b-a561-26d4b63f8c82 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de28bb95bc464874a7f70b33e5c5f8e3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:49.409081 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-05ee7315-ac71-436b-a561-26d4b63f8c82 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de28bb95bc464874a7f70b33e5c5f8e3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:49.422541 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-05ee7315-ac71-436b-a561-26d4b63f8c82 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de28bb95bc464874a7f70b33e5c5f8e3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:49.423183 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-05ee7315-ac71-436b-a561-26d4b63f8c82 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de28bb95bc464874a7f70b33e5c5f8e3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:49.425133 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:11:49.430858 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-05ee7315-ac71-436b-a561-26d4b63f8c82 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de28bb95bc464874a7f70b33e5c5f8e3'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:49.436926 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1404/2790] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:11:49 2022] DELETE /identity/v3/users/de28bb95bc464874a7f70b33e5c5f8e3 => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:11:49.699039 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c52cbf1-8597-4cb6-906f-0a134453717c admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:11:49.700469 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3c52cbf1-8597-4cb6-906f-0a134453717c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:11:49.715133 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c52cbf1-8597-4cb6-906f-0a134453717c admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:11:49.774149 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43bfbcf0-82ef-4b78-ba14-cdef63fe7ffe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:11:49.774811 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bfbcf0-82ef-4b78-ba14-cdef63fe7ffe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:11:49.775063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bfbcf0-82ef-4b78-ba14-cdef63fe7ffe admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:11:49.776048 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bfbcf0-82ef-4b78-ba14-cdef63fe7ffe admin admin] PATH_INFO: `/v3/projects/f2789f9cb0bd44aaa264d2e76944cbc9` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:11:49.777135 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43bfbcf0-82ef-4b78-ba14-cdef63fe7ffe admin admin] RBAC: Authorizing `identity:delete_project(project_id=f2789f9cb0bd44aaa264d2e76944cbc9)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:11:49.779111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43bfbcf0-82ef-4b78-ba14-cdef63fe7ffe admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:11:49.894946 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-43bfbcf0-82ef-4b78-ba14-cdef63fe7ffe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f2789f9cb0bd44aaa264d2e76944cbc9', 'request_id': 'req-43bfbcf0-82ef-4b78-ba14-cdef63fe7ffe'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:49.971861 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-43bfbcf0-82ef-4b78-ba14-cdef63fe7ffe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f2789f9cb0bd44aaa264d2e76944cbc9', 'request_id': 'req-43bfbcf0-82ef-4b78-ba14-cdef63fe7ffe'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:49.978318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-43bfbcf0-82ef-4b78-ba14-cdef63fe7ffe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f2789f9cb0bd44aaa264d2e76944cbc9', 'request_id': 'req-43bfbcf0-82ef-4b78-ba14-cdef63fe7ffe'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:49.990315 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-43bfbcf0-82ef-4b78-ba14-cdef63fe7ffe admin admin] The token cache is being invalidate because project f2789f9cb0bd44aaa264d2e76944cbc9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:11:49.990632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-43bfbcf0-82ef-4b78-ba14-cdef63fe7ffe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:49.991272 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-43bfbcf0-82ef-4b78-ba14-cdef63fe7ffe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:11:49.992789 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1387/2791] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:11:49 2022] DELETE /identity/v3/projects/f2789f9cb0bd44aaa264d2e76944cbc9 => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:11:59.543149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=4 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:11:59.987206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:DELETE=10,keystone:SELECT=57,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:12:09.696677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ee7315-ac71-436b-a561-26d4b63f8c82 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:12:09.698116 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-05ee7315-ac71-436b-a561-26d4b63f8c82 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:12:09.757224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ee7315-ac71-436b-a561-26d4b63f8c82 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:12:09.807083 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e6e29f-ab82-44b2-99f7-5b77d71bf5fc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:12:09.807687 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e6e29f-ab82-44b2-99f7-5b77d71bf5fc service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:12:09.807965 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e6e29f-ab82-44b2-99f7-5b77d71bf5fc service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:12:09.808234 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e6e29f-ab82-44b2-99f7-5b77d71bf5fc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:12:09.810272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-73e6e29f-ab82-44b2-99f7-5b77d71bf5fc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:12:09.880685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e6e29f-ab82-44b2-99f7-5b77d71bf5fc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:12:09.882109 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e6e29f-ab82-44b2-99f7-5b77d71bf5fc service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:12:09.893586 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1405/2792] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:12:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:12:11.073277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43bfbcf0-82ef-4b78-ba14-cdef63fe7ffe admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:12:11.074994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-43bfbcf0-82ef-4b78-ba14-cdef63fe7ffe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:12:11.087246 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43bfbcf0-82ef-4b78-ba14-cdef63fe7ffe admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:12:11.091705 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb33a2f8-d5a4-4e0c-8b29-19bd96c72f17 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:12:11.092303 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb33a2f8-d5a4-4e0c-8b29-19bd96c72f17 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:12:11.092580 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb33a2f8-d5a4-4e0c-8b29-19bd96c72f17 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:12:11.092849 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb33a2f8-d5a4-4e0c-8b29-19bd96c72f17 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:12:11.094586 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-fb33a2f8-d5a4-4e0c-8b29-19bd96c72f17 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:12:11.108795 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb33a2f8-d5a4-4e0c-8b29-19bd96c72f17 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:12:11.110257 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb33a2f8-d5a4-4e0c-8b29-19bd96c72f17 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:12:11.121443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1388/2793] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:12:11 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:12:19.888667 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=74 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:12:21.116316 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=6 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:12:59.281416 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e6e29f-ab82-44b2-99f7-5b77d71bf5fc service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:12:59.283705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-73e6e29f-ab82-44b2-99f7-5b77d71bf5fc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:12:59.294756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e6e29f-ab82-44b2-99f7-5b77d71bf5fc service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:12:59.298390 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f769fece-78ac-4e5a-b22e-e4825826c241 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:12:59.298984 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f769fece-78ac-4e5a-b22e-e4825826c241 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:12:59.299259 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f769fece-78ac-4e5a-b22e-e4825826c241 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:12:59.299525 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f769fece-78ac-4e5a-b22e-e4825826c241 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:12:59.301332 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f769fece-78ac-4e5a-b22e-e4825826c241 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:12:59.311551 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f769fece-78ac-4e5a-b22e-e4825826c241 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:12:59.312967 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f769fece-78ac-4e5a-b22e-e4825826c241 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:12:59.321466 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1406/2794] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:12:59 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:13:09.317734 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=6 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:13:38.075281 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb33a2f8-d5a4-4e0c-8b29-19bd96c72f17 service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:13:38.076853 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-fb33a2f8-d5a4-4e0c-8b29-19bd96c72f17 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:13:38.091287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb33a2f8-d5a4-4e0c-8b29-19bd96c72f17 service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:13:38.096865 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7904b928-5515-4478-bf2c-3d4ae3471dde service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:13:38.098080 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7904b928-5515-4478-bf2c-3d4ae3471dde service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:13:38.098383 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7904b928-5515-4478-bf2c-3d4ae3471dde service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:13:38.098663 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7904b928-5515-4478-bf2c-3d4ae3471dde service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:13:38.100374 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-7904b928-5515-4478-bf2c-3d4ae3471dde service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:13:38.113839 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7904b928-5515-4478-bf2c-3d4ae3471dde service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:13:38.115757 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7904b928-5515-4478-bf2c-3d4ae3471dde service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:13:38.127634 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1389/2795] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:13:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:13:43.677298 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288dfea8-62f3-4406-911d-f766557370cb None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:13:43.677840 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288dfea8-62f3-4406-911d-f766557370cb None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:13:43.677958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288dfea8-62f3-4406-911d-f766557370cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:13:43.697902 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-288dfea8-62f3-4406-911d-f766557370cb None None] MFA Rules not processed for user `1e18b5b70f8b4939bebae2d7ef261740`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:13:43.699111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-288dfea8-62f3-4406-911d-f766557370cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:13:43.752964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1407/2796] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:13:43 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:13:44.050762 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7904b928-5515-4478-bf2c-3d4ae3471dde service nova] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:13:44.061241 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7904b928-5515-4478-bf2c-3d4ae3471dde service nova] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:13:44.064751 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9baa27-8e88-4efb-9aef-15c0a6cddacb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:13:44.065346 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9baa27-8e88-4efb-9aef-15c0a6cddacb service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:13:44.065622 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9baa27-8e88-4efb-9aef-15c0a6cddacb service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:13:44.065888 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9baa27-8e88-4efb-9aef-15c0a6cddacb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:13:44.067629 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ff9baa27-8e88-4efb-9aef-15c0a6cddacb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:13:44.077007 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9baa27-8e88-4efb-9aef-15c0a6cddacb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:13:44.078436 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9baa27-8e88-4efb-9aef-15c0a6cddacb service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:13:44.086736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1390/2797] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:13:44 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:13:44.486848 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288dfea8-62f3-4406-911d-f766557370cb None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:13:44.488356 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-288dfea8-62f3-4406-911d-f766557370cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:13:44.564865 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288dfea8-62f3-4406-911d-f766557370cb None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:13:44.570216 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6098232-2693-497f-b890-c30c1ceadafc service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce249d4a99c34891b6ba941f57a14dd4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:13:44.570875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6098232-2693-497f-b890-c30c1ceadafc service placement] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:13:44.571158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6098232-2693-497f-b890-c30c1ceadafc service placement] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:13:44.571426 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6098232-2693-497f-b890-c30c1ceadafc service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:13:44.573205 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f6098232-2693-497f-b890-c30c1ceadafc service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:13:44.585141 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6098232-2693-497f-b890-c30c1ceadafc service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:13:44.586720 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6098232-2693-497f-b890-c30c1ceadafc service placement] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:13:44.596549 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1408/2798] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:13:44 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:13:53.909121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9baa27-8e88-4efb-9aef-15c0a6cddacb service neutron] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:13:53.923869 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9baa27-8e88-4efb-9aef-15c0a6cddacb service neutron] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:13:53.929605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f52186-b901-40e5-add7-05411febf54d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:13:53.930205 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f52186-b901-40e5-add7-05411febf54d service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:13:53.930558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f52186-b901-40e5-add7-05411febf54d service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:13:53.942121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f52186-b901-40e5-add7-05411febf54d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:13:53.942121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-36f52186-b901-40e5-add7-05411febf54d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:13:53.945192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f52186-b901-40e5-add7-05411febf54d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:13:53.947171 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f52186-b901-40e5-add7-05411febf54d service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:13:53.958574 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1391/2799] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:13:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:13:54.592361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=57 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:13:57.433068 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6098232-2693-497f-b890-c30c1ceadafc service placement] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:13:57.434646 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-f6098232-2693-497f-b890-c30c1ceadafc service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:13:57.450384 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6098232-2693-497f-b890-c30c1ceadafc service placement] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:13:57.456227 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec02ac8-3874-45fd-8c87-2942eca77537 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:13:57.456849 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec02ac8-3874-45fd-8c87-2942eca77537 service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:13:57.457129 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec02ac8-3874-45fd-8c87-2942eca77537 service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:13:57.457405 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec02ac8-3874-45fd-8c87-2942eca77537 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:13:57.459212 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-eec02ac8-3874-45fd-8c87-2942eca77537 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:13:57.472770 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec02ac8-3874-45fd-8c87-2942eca77537 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:13:57.474451 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec02ac8-3874-45fd-8c87-2942eca77537 service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:13:57.486875 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1409/2800] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:13:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:14:03.953397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=18 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:14:07.481959 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=6 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:15:13.601619 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f52186-b901-40e5-add7-05411febf54d service nova] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:13.616016 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f52186-b901-40e5-add7-05411febf54d service nova] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:13.622005 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3289bd98-2d10-47a6-920b-4f8a7ac03ffc service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce249d4a99c34891b6ba941f57a14dd4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:13.622647 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3289bd98-2d10-47a6-920b-4f8a7ac03ffc service placement] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:13.622929 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3289bd98-2d10-47a6-920b-4f8a7ac03ffc service placement] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:13.623233 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3289bd98-2d10-47a6-920b-4f8a7ac03ffc service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:13.625002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3289bd98-2d10-47a6-920b-4f8a7ac03ffc service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:13.637314 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3289bd98-2d10-47a6-920b-4f8a7ac03ffc service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:13.638962 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3289bd98-2d10-47a6-920b-4f8a7ac03ffc service placement] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:13.649406 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1392/2801] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:15:13 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:15:13.735868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec02ac8-3874-45fd-8c87-2942eca77537 service nova] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:13.737277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-eec02ac8-3874-45fd-8c87-2942eca77537 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:13.808081 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec02ac8-3874-45fd-8c87-2942eca77537 service nova] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:13.813004 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39319fde-d56c-4854-9a0b-8e07746be28f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:13.813624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39319fde-d56c-4854-9a0b-8e07746be28f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:13.813904 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39319fde-d56c-4854-9a0b-8e07746be28f admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:13.814172 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39319fde-d56c-4854-9a0b-8e07746be28f admin admin] PATH_INFO: `/v3/users/1e18b5b70f8b4939bebae2d7ef261740` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:13.815341 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39319fde-d56c-4854-9a0b-8e07746be28f admin admin] RBAC: Authorizing `identity:delete_user(user_id=1e18b5b70f8b4939bebae2d7ef261740)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:13.817249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39319fde-d56c-4854-9a0b-8e07746be28f admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:13.947262 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-39319fde-d56c-4854-9a0b-8e07746be28f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1e18b5b70f8b4939bebae2d7ef261740'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:14.029881 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-39319fde-d56c-4854-9a0b-8e07746be28f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1e18b5b70f8b4939bebae2d7ef261740'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:14.042714 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-39319fde-d56c-4854-9a0b-8e07746be28f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1e18b5b70f8b4939bebae2d7ef261740'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:14.043370 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-39319fde-d56c-4854-9a0b-8e07746be28f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1e18b5b70f8b4939bebae2d7ef261740'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:14.050661 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-39319fde-d56c-4854-9a0b-8e07746be28f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1e18b5b70f8b4939bebae2d7ef261740'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:14.056503 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1410/2802] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:15:13 2022] DELETE /identity/v3/users/1e18b5b70f8b4939bebae2d7ef261740 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:15:14.302131 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3289bd98-2d10-47a6-920b-4f8a7ac03ffc service placement] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:14.313845 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3289bd98-2d10-47a6-920b-4f8a7ac03ffc service placement] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:14.374898 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97344ab1-0e68-4486-91e7-608a58e86c34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:14.375530 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97344ab1-0e68-4486-91e7-608a58e86c34 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:14.375809 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97344ab1-0e68-4486-91e7-608a58e86c34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:14.376078 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97344ab1-0e68-4486-91e7-608a58e86c34 admin admin] PATH_INFO: `/v3/projects/403c04c19c0c45398d6b232c03a4b7a6` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:14.377182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97344ab1-0e68-4486-91e7-608a58e86c34 admin admin] RBAC: Authorizing `identity:delete_project(project_id=403c04c19c0c45398d6b232c03a4b7a6)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:14.379120 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97344ab1-0e68-4486-91e7-608a58e86c34 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:14.471986 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-97344ab1-0e68-4486-91e7-608a58e86c34 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '403c04c19c0c45398d6b232c03a4b7a6', 'request_id': 'req-97344ab1-0e68-4486-91e7-608a58e86c34'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:14.568343 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-97344ab1-0e68-4486-91e7-608a58e86c34 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '403c04c19c0c45398d6b232c03a4b7a6', 'request_id': 'req-97344ab1-0e68-4486-91e7-608a58e86c34'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:14.574155 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-97344ab1-0e68-4486-91e7-608a58e86c34 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '403c04c19c0c45398d6b232c03a4b7a6', 'request_id': 'req-97344ab1-0e68-4486-91e7-608a58e86c34'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:14.586717 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-97344ab1-0e68-4486-91e7-608a58e86c34 admin admin] The token cache is being invalidate because project 403c04c19c0c45398d6b232c03a4b7a6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:15:14.587049 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-97344ab1-0e68-4486-91e7-608a58e86c34 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:14.587677 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-97344ab1-0e68-4486-91e7-608a58e86c34 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:14.589147 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1393/2803] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:15:14 2022] DELETE /identity/v3/projects/403c04c19c0c45398d6b232c03a4b7a6 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:15:14.594249 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09647a86-56c8-43ad-b8c4-9d4f558da31e None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:14.594571 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09647a86-56c8-43ad-b8c4-9d4f558da31e None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:14.594847 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09647a86-56c8-43ad-b8c4-9d4f558da31e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:14.616844 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-09647a86-56c8-43ad-b8c4-9d4f558da31e None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:15:14.618055 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-09647a86-56c8-43ad-b8c4-9d4f558da31e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:14.676865 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1411/2804] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:15:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:15:14.685635 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92ee300-1625-4609-8721-8b80e4a5386c None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:14.689185 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92ee300-1625-4609-8721-8b80e4a5386c None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:14.689185 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92ee300-1625-4609-8721-8b80e4a5386c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:14.707177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-f92ee300-1625-4609-8721-8b80e4a5386c None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:15:14.708371 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-f92ee300-1625-4609-8721-8b80e4a5386c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:14.716371 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1394/2805] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:15:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:15:14.720538 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09647a86-56c8-43ad-b8c4-9d4f558da31e None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:14.730396 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09647a86-56c8-43ad-b8c4-9d4f558da31e None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:14.734930 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb8aab2-0b5a-4681-b107-bbe34c3b698b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:14.735520 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb8aab2-0b5a-4681-b107-bbe34c3b698b admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:14.735798 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb8aab2-0b5a-4681-b107-bbe34c3b698b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:14.736067 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb8aab2-0b5a-4681-b107-bbe34c3b698b admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:14.736705 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb8aab2-0b5a-4681-b107-bbe34c3b698b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:15:14.737003 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb8aab2-0b5a-4681-b107-bbe34c3b698b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:14.738624 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb8aab2-0b5a-4681-b107-bbe34c3b698b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:14.751045 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1412/2806] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:15:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:15:14.755362 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92ee300-1625-4609-8721-8b80e4a5386c None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:14.766902 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92ee300-1625-4609-8721-8b80e4a5386c None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:14.772355 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55a2b29-a57d-4c83-b8c8-a7f1f7f96545 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:14.772953 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55a2b29-a57d-4c83-b8c8-a7f1f7f96545 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:14.773231 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55a2b29-a57d-4c83-b8c8-a7f1f7f96545 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:14.773494 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55a2b29-a57d-4c83-b8c8-a7f1f7f96545 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:14.774111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55a2b29-a57d-4c83-b8c8-a7f1f7f96545 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:14.776040 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55a2b29-a57d-4c83-b8c8-a7f1f7f96545 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:14.874584 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1395/2807] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:15:14 2022] POST /identity/v3/projects => generated 377 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:15:14.878689 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb8aab2-0b5a-4681-b107-bbe34c3b698b admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:14.890250 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb8aab2-0b5a-4681-b107-bbe34c3b698b admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:14.893824 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:15:14.947689 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aefd199-06fe-4b3c-b1d4-e55772a80128 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:14.948293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aefd199-06fe-4b3c-b1d4-e55772a80128 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:14.948569 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aefd199-06fe-4b3c-b1d4-e55772a80128 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:14.948836 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aefd199-06fe-4b3c-b1d4-e55772a80128 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:14.949470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aefd199-06fe-4b3c-b1d4-e55772a80128 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:14.951567 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aefd199-06fe-4b3c-b1d4-e55772a80128 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:15.058219 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1413/2808] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:15:14 2022] POST /identity/v3/users => generated 289 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:15:15.060226 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55a2b29-a57d-4c83-b8c8-a7f1f7f96545 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:15.070482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55a2b29-a57d-4c83-b8c8-a7f1f7f96545 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:15.075143 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55b5c40-ca44-4436-b821-febb2345faf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:15.075738 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55b5c40-ca44-4436-b821-febb2345faf2 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:15.076014 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55b5c40-ca44-4436-b821-febb2345faf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:15.076286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55b5c40-ca44-4436-b821-febb2345faf2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:15.076923 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55b5c40-ca44-4436-b821-febb2345faf2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:15.078556 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55b5c40-ca44-4436-b821-febb2345faf2 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:15.088305 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1396/2809] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:15:15 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:15:15.092384 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aefd199-06fe-4b3c-b1d4-e55772a80128 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:15.103636 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aefd199-06fe-4b3c-b1d4-e55772a80128 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:15.110971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e155e2bf-dc96-48c2-94da-c7505b548a69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:15.111577 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e155e2bf-dc96-48c2-94da-c7505b548a69 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:15.111854 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e155e2bf-dc96-48c2-94da-c7505b548a69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:15.112125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e155e2bf-dc96-48c2-94da-c7505b548a69 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:15.112762 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e155e2bf-dc96-48c2-94da-c7505b548a69 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:15.114366 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e155e2bf-dc96-48c2-94da-c7505b548a69 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:15.126896 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1414/2810] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:15:15 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:15:15.131127 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55b5c40-ca44-4436-b821-febb2345faf2 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:15.141462 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55b5c40-ca44-4436-b821-febb2345faf2 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:15.146266 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eade934-d726-4864-9320-8c914426c528 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:15.146942 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eade934-d726-4864-9320-8c914426c528 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:15.147221 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eade934-d726-4864-9320-8c914426c528 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:15.147515 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eade934-d726-4864-9320-8c914426c528 admin admin] PATH_INFO: `/v3/projects/89d0b8db43124fd796679613af3b03d7/users/fa9c2a61162941e099ce1fc96d89f75f/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:15.163065 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eade934-d726-4864-9320-8c914426c528 admin admin] RBAC: Authorizing `identity:create_grant(project_id=89d0b8db43124fd796679613af3b03d7, user_id=fa9c2a61162941e099ce1fc96d89f75f, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:15.165673 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eade934-d726-4864-9320-8c914426c528 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:15.232350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1397/2811] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:15:15 2022] PUT /identity/v3/projects/89d0b8db43124fd796679613af3b03d7/users/fa9c2a61162941e099ce1fc96d89f75f/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:15:15.242788 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7c30fa-7133-4641-b3a1-132a70a7c3c8 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:15.243078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7c30fa-7133-4641-b3a1-132a70a7c3c8 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:15.243351 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7c30fa-7133-4641-b3a1-132a70a7c3c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:15.272738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-2f7c30fa-7133-4641-b3a1-132a70a7c3c8 None None] MFA Rules not processed for user `fa9c2a61162941e099ce1fc96d89f75f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:15:15.274015 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-2f7c30fa-7133-4641-b3a1-132a70a7c3c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:15.340765 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1415/2812] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:15:15 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:15:15.345799 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eade934-d726-4864-9320-8c914426c528 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:15.351403 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:15:15.408009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eade934-d726-4864-9320-8c914426c528 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:15.412718 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e68f89-928d-4c42-b6c4-46be22192882 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:15.413564 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e68f89-928d-4c42-b6c4-46be22192882 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:15.414025 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e68f89-928d-4c42-b6c4-46be22192882 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:15.414358 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e68f89-928d-4c42-b6c4-46be22192882 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:15.415399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e68f89-928d-4c42-b6c4-46be22192882 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:15.417287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e68f89-928d-4c42-b6c4-46be22192882 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:15.511210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1398/2813] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:15:15 2022] POST /identity/v3/projects => generated 379 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:15:15.515817 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7c30fa-7133-4641-b3a1-132a70a7c3c8 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:15.526932 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7c30fa-7133-4641-b3a1-132a70a7c3c8 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:15.583589 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd40c60-0df9-4f28-9018-64c0356610ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:15.584189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd40c60-0df9-4f28-9018-64c0356610ee admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:15.584465 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd40c60-0df9-4f28-9018-64c0356610ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:15.584733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd40c60-0df9-4f28-9018-64c0356610ee admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:15.585352 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd40c60-0df9-4f28-9018-64c0356610ee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:15.587307 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd40c60-0df9-4f28-9018-64c0356610ee admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:15.688413 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1416/2814] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:15:15 2022] POST /identity/v3/users => generated 290 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:15:15.692417 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e68f89-928d-4c42-b6c4-46be22192882 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:15.703774 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e68f89-928d-4c42-b6c4-46be22192882 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:15.708429 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78a03a5-5401-45b3-ae81-2070a8f09972 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:15.709025 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78a03a5-5401-45b3-ae81-2070a8f09972 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:15.709307 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78a03a5-5401-45b3-ae81-2070a8f09972 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:15.709582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78a03a5-5401-45b3-ae81-2070a8f09972 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:15.710219 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78a03a5-5401-45b3-ae81-2070a8f09972 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:15.711837 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78a03a5-5401-45b3-ae81-2070a8f09972 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:15.723445 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1399/2815] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:15:15 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:15:15.727072 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd40c60-0df9-4f28-9018-64c0356610ee admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:15.733285 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:15:15.737131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd40c60-0df9-4f28-9018-64c0356610ee admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:15.741738 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27b4495-2a76-48c1-920b-bde6326e0627 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:15.746710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27b4495-2a76-48c1-920b-bde6326e0627 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:15.746710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27b4495-2a76-48c1-920b-bde6326e0627 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:15.746710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27b4495-2a76-48c1-920b-bde6326e0627 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:15.746710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27b4495-2a76-48c1-920b-bde6326e0627 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:15.746710 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27b4495-2a76-48c1-920b-bde6326e0627 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:15.756889 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1417/2816] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:15:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:15:15.760912 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78a03a5-5401-45b3-ae81-2070a8f09972 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:15.771545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78a03a5-5401-45b3-ae81-2070a8f09972 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:15.776098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991b9e89-28e5-46e2-a7b3-977ea56e3e34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:15.776738 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991b9e89-28e5-46e2-a7b3-977ea56e3e34 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:15.777011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991b9e89-28e5-46e2-a7b3-977ea56e3e34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:15.777280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991b9e89-28e5-46e2-a7b3-977ea56e3e34 admin admin] PATH_INFO: `/v3/projects/dcb3b6b73a0a4509b277b576320e0eee/users/44e8c9a3ea934c1c96f9c477a8343da6/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:15.792650 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991b9e89-28e5-46e2-a7b3-977ea56e3e34 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dcb3b6b73a0a4509b277b576320e0eee, user_id=44e8c9a3ea934c1c96f9c477a8343da6, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:15.795446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991b9e89-28e5-46e2-a7b3-977ea56e3e34 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:15.911154 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1400/2817] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:15:15 2022] PUT /identity/v3/projects/dcb3b6b73a0a4509b277b576320e0eee/users/44e8c9a3ea934c1c96f9c477a8343da6/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:15:15.923021 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6327a99-c44e-46ec-856c-c474c02de153 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:15.923308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6327a99-c44e-46ec-856c-c474c02de153 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:15.927722 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6327a99-c44e-46ec-856c-c474c02de153 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:15.952420 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-c6327a99-c44e-46ec-856c-c474c02de153 None None] MFA Rules not processed for user `44e8c9a3ea934c1c96f9c477a8343da6`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:15:15.953637 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-c6327a99-c44e-46ec-856c-c474c02de153 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:16.016087 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1418/2818] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:15:15 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:15:16.020526 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991b9e89-28e5-46e2-a7b3-977ea56e3e34 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:16.080775 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991b9e89-28e5-46e2-a7b3-977ea56e3e34 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:16.085389 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651f4682-a4e8-4364-8bb6-84388d553b79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:16.085984 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651f4682-a4e8-4364-8bb6-84388d553b79 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:16.086284 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651f4682-a4e8-4364-8bb6-84388d553b79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:16.086581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651f4682-a4e8-4364-8bb6-84388d553b79 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:16.087195 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651f4682-a4e8-4364-8bb6-84388d553b79 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:16.089086 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651f4682-a4e8-4364-8bb6-84388d553b79 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:16.176449 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1401/2819] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:15:16 2022] POST /identity/v3/projects => generated 379 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:15:16.181887 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6327a99-c44e-46ec-856c-c474c02de153 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:16.192483 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6327a99-c44e-46ec-856c-c474c02de153 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:16.252174 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8547a8-ccab-4578-9e30-dcc213859efa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:16.252827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8547a8-ccab-4578-9e30-dcc213859efa admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:16.253110 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8547a8-ccab-4578-9e30-dcc213859efa admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:16.253387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8547a8-ccab-4578-9e30-dcc213859efa admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:16.254024 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8547a8-ccab-4578-9e30-dcc213859efa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:16.256017 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8547a8-ccab-4578-9e30-dcc213859efa admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:16.333193 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1419/2820] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:15:16 2022] POST /identity/v3/users => generated 296 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:15:16.337546 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651f4682-a4e8-4364-8bb6-84388d553b79 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:16.348083 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651f4682-a4e8-4364-8bb6-84388d553b79 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:16.352873 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b631263-4a3d-4726-9ef4-4d164912d964 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:16.353495 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b631263-4a3d-4726-9ef4-4d164912d964 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:16.353777 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b631263-4a3d-4726-9ef4-4d164912d964 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:16.354051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b631263-4a3d-4726-9ef4-4d164912d964 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:16.354761 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b631263-4a3d-4726-9ef4-4d164912d964 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:16.356857 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b631263-4a3d-4726-9ef4-4d164912d964 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:16.367404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1402/2821] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:15:16 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:15:16.375830 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8547a8-ccab-4578-9e30-dcc213859efa admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:16.386021 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8547a8-ccab-4578-9e30-dcc213859efa admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:16.390964 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca54d6c-9c06-494b-9418-d94feca069c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:16.391606 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca54d6c-9c06-494b-9418-d94feca069c8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:16.391893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca54d6c-9c06-494b-9418-d94feca069c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:16.392161 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca54d6c-9c06-494b-9418-d94feca069c8 admin admin] PATH_INFO: `/v3/projects/87b5a1ebba1242c4a1ad1373cf097022/users/452094ea3ec44c31a6b6ce039e3f6edd/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:16.407361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca54d6c-9c06-494b-9418-d94feca069c8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=87b5a1ebba1242c4a1ad1373cf097022, user_id=452094ea3ec44c31a6b6ce039e3f6edd, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:16.410218 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca54d6c-9c06-494b-9418-d94feca069c8 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:16.477960 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1420/2822] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:15:16 2022] PUT /identity/v3/projects/87b5a1ebba1242c4a1ad1373cf097022/users/452094ea3ec44c31a6b6ce039e3f6edd/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:15:16.482439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b631263-4a3d-4726-9ef4-4d164912d964 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:16.492252 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b631263-4a3d-4726-9ef4-4d164912d964 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:16.519077 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:15:16.548891 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be9f6d25-1344-487c-b902-612e77819c4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:16.549489 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9f6d25-1344-487c-b902-612e77819c4a admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:16.549769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9f6d25-1344-487c-b902-612e77819c4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:16.550069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be9f6d25-1344-487c-b902-612e77819c4a admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:16.550732 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9f6d25-1344-487c-b902-612e77819c4a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:16.552306 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be9f6d25-1344-487c-b902-612e77819c4a admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:16.562841 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1403/2823] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:15:16 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:15:16.566550 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca54d6c-9c06-494b-9418-d94feca069c8 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:16.577695 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca54d6c-9c06-494b-9418-d94feca069c8 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:16.582281 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5962f080-c8a3-4e86-ba39-e50a506b30ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:16.582954 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5962f080-c8a3-4e86-ba39-e50a506b30ef admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:16.583226 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5962f080-c8a3-4e86-ba39-e50a506b30ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:16.583492 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5962f080-c8a3-4e86-ba39-e50a506b30ef admin admin] PATH_INFO: `/v3/projects/87b5a1ebba1242c4a1ad1373cf097022/users/452094ea3ec44c31a6b6ce039e3f6edd/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:16.585421 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5962f080-c8a3-4e86-ba39-e50a506b30ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=87b5a1ebba1242c4a1ad1373cf097022, user_id=452094ea3ec44c31a6b6ce039e3f6edd, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:16.588054 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5962f080-c8a3-4e86-ba39-e50a506b30ef admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:16.598653 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1421/2824] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:15:16 2022] PUT /identity/v3/projects/87b5a1ebba1242c4a1ad1373cf097022/users/452094ea3ec44c31a6b6ce039e3f6edd/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:15:16.609773 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76459ae3-dd3d-4a5f-843f-855542082567 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:16.610063 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76459ae3-dd3d-4a5f-843f-855542082567 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:16.610353 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76459ae3-dd3d-4a5f-843f-855542082567 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:16.639269 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-76459ae3-dd3d-4a5f-843f-855542082567 None None] MFA Rules not processed for user `452094ea3ec44c31a6b6ce039e3f6edd`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:15:16.640536 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-76459ae3-dd3d-4a5f-843f-855542082567 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:16.706309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1404/2825] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:15:16 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:15:16.718234 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5962f080-c8a3-4e86-ba39-e50a506b30ef admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:16.719920 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-5962f080-c8a3-4e86-ba39-e50a506b30ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:16.759273 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:15:16.775985 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5962f080-c8a3-4e86-ba39-e50a506b30ef admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:16.780132 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de823368-7fa6-45eb-b4af-6e7eb884e32d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:16.780727 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de823368-7fa6-45eb-b4af-6e7eb884e32d service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:16.781000 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de823368-7fa6-45eb-b4af-6e7eb884e32d service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:16.781266 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de823368-7fa6-45eb-b4af-6e7eb884e32d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:16.790109 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de823368-7fa6-45eb-b4af-6e7eb884e32d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:16.791551 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de823368-7fa6-45eb-b4af-6e7eb884e32d service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:16.801968 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1422/2826] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:15:16 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:15:16.909395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76459ae3-dd3d-4a5f-843f-855542082567 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:16.910835 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-76459ae3-dd3d-4a5f-843f-855542082567 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:16.923989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76459ae3-dd3d-4a5f-843f-855542082567 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:16.927610 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7dd9cdb-94a8-4cfa-9ff8-e6ad7b076de1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:16.928206 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dd9cdb-94a8-4cfa-9ff8-e6ad7b076de1 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:16.928501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dd9cdb-94a8-4cfa-9ff8-e6ad7b076de1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:16.928754 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7dd9cdb-94a8-4cfa-9ff8-e6ad7b076de1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:16.985395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7dd9cdb-94a8-4cfa-9ff8-e6ad7b076de1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:16.986860 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7dd9cdb-94a8-4cfa-9ff8-e6ad7b076de1 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:16.995918 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1405/2827] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:15:16 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:15:20.919912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de823368-7fa6-45eb-b4af-6e7eb884e32d service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:20.982646 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de823368-7fa6-45eb-b4af-6e7eb884e32d service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:20.987183 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca0c9f6-c6b5-4aae-a0d2-3a2c7c8e7f13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:20.987802 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca0c9f6-c6b5-4aae-a0d2-3a2c7c8e7f13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:20.988103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca0c9f6-c6b5-4aae-a0d2-3a2c7c8e7f13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:20.988374 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca0c9f6-c6b5-4aae-a0d2-3a2c7c8e7f13 admin admin] PATH_INFO: `/v3/users/fa9c2a61162941e099ce1fc96d89f75f` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:20.989518 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca0c9f6-c6b5-4aae-a0d2-3a2c7c8e7f13 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fa9c2a61162941e099ce1fc96d89f75f)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:20.991470 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca0c9f6-c6b5-4aae-a0d2-3a2c7c8e7f13 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:21.183170 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0ca0c9f6-c6b5-4aae-a0d2-3a2c7c8e7f13 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fa9c2a61162941e099ce1fc96d89f75f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:21.271677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0ca0c9f6-c6b5-4aae-a0d2-3a2c7c8e7f13 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fa9c2a61162941e099ce1fc96d89f75f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:21.285602 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0ca0c9f6-c6b5-4aae-a0d2-3a2c7c8e7f13 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fa9c2a61162941e099ce1fc96d89f75f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:21.286243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0ca0c9f6-c6b5-4aae-a0d2-3a2c7c8e7f13 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fa9c2a61162941e099ce1fc96d89f75f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:21.293324 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0ca0c9f6-c6b5-4aae-a0d2-3a2c7c8e7f13 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fa9c2a61162941e099ce1fc96d89f75f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:21.300679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1423/2828] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:15:20 2022] DELETE /identity/v3/users/fa9c2a61162941e099ce1fc96d89f75f => generated 0 bytes in 381 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:15:21.305193 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8936081f-5107-48e2-bb48-f8b11e7ddbaf None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:21.305656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8936081f-5107-48e2-bb48-f8b11e7ddbaf None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:21.305778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8936081f-5107-48e2-bb48-f8b11e7ddbaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:21.326658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-8936081f-5107-48e2-bb48-f8b11e7ddbaf None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:15:21.327888 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8936081f-5107-48e2-bb48-f8b11e7ddbaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:21.373728 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=100 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:15:21.388916 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1406/2829] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:15:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:15:21.398764 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca0c9f6-c6b5-4aae-a0d2-3a2c7c8e7f13 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:21.400206 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-0ca0c9f6-c6b5-4aae-a0d2-3a2c7c8e7f13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:21.459292 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca0c9f6-c6b5-4aae-a0d2-3a2c7c8e7f13 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:21.462990 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d99c3dd-fcf5-4901-a9d7-3023739c61bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:21.463593 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d99c3dd-fcf5-4901-a9d7-3023739c61bf service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:21.463871 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d99c3dd-fcf5-4901-a9d7-3023739c61bf service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:21.464140 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d99c3dd-fcf5-4901-a9d7-3023739c61bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:21.474127 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d99c3dd-fcf5-4901-a9d7-3023739c61bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:21.475592 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d99c3dd-fcf5-4901-a9d7-3023739c61bf service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:21.480510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:15:21.486178 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1424/2830] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:15:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:15:21.907096 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8936081f-5107-48e2-bb48-f8b11e7ddbaf None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:21.908517 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8936081f-5107-48e2-bb48-f8b11e7ddbaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:21.922177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8936081f-5107-48e2-bb48-f8b11e7ddbaf None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:21.926791 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78f0c76-4ff0-4c6d-a56e-472836561f81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:21.927413 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78f0c76-4ff0-4c6d-a56e-472836561f81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:21.927688 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78f0c76-4ff0-4c6d-a56e-472836561f81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:21.927957 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78f0c76-4ff0-4c6d-a56e-472836561f81 admin admin] PATH_INFO: `/v3/projects/89d0b8db43124fd796679613af3b03d7` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:21.929062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78f0c76-4ff0-4c6d-a56e-472836561f81 admin admin] RBAC: Authorizing `identity:delete_project(project_id=89d0b8db43124fd796679613af3b03d7)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:21.931011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78f0c76-4ff0-4c6d-a56e-472836561f81 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:22.070882 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d78f0c76-4ff0-4c6d-a56e-472836561f81 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '89d0b8db43124fd796679613af3b03d7', 'request_id': 'req-d78f0c76-4ff0-4c6d-a56e-472836561f81'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:22.189991 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d78f0c76-4ff0-4c6d-a56e-472836561f81 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '89d0b8db43124fd796679613af3b03d7', 'request_id': 'req-d78f0c76-4ff0-4c6d-a56e-472836561f81'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:22.196330 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d78f0c76-4ff0-4c6d-a56e-472836561f81 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '89d0b8db43124fd796679613af3b03d7', 'request_id': 'req-d78f0c76-4ff0-4c6d-a56e-472836561f81'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:22.208473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d78f0c76-4ff0-4c6d-a56e-472836561f81 admin admin] The token cache is being invalidate because project 89d0b8db43124fd796679613af3b03d7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:15:22.208759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d78f0c76-4ff0-4c6d-a56e-472836561f81 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:22.209402 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-d78f0c76-4ff0-4c6d-a56e-472836561f81 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:22.210928 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1407/2831] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:15:21 2022] DELETE /identity/v3/projects/89d0b8db43124fd796679613af3b03d7 => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:15:22.215235 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d99c3dd-fcf5-4901-a9d7-3023739c61bf service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:22.225104 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d99c3dd-fcf5-4901-a9d7-3023739c61bf service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:22.281381 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36366fb9-9c39-4472-9de2-33435fc76279 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:22.281996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36366fb9-9c39-4472-9de2-33435fc76279 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:22.282756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36366fb9-9c39-4472-9de2-33435fc76279 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:22.282756 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36366fb9-9c39-4472-9de2-33435fc76279 admin admin] PATH_INFO: `/v3/users/44e8c9a3ea934c1c96f9c477a8343da6` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:22.283694 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36366fb9-9c39-4472-9de2-33435fc76279 admin admin] RBAC: Authorizing `identity:delete_user(user_id=44e8c9a3ea934c1c96f9c477a8343da6)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:22.285602 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36366fb9-9c39-4472-9de2-33435fc76279 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:22.457329 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-36366fb9-9c39-4472-9de2-33435fc76279 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '44e8c9a3ea934c1c96f9c477a8343da6'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:22.511381 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-36366fb9-9c39-4472-9de2-33435fc76279 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '44e8c9a3ea934c1c96f9c477a8343da6'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:22.524504 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-36366fb9-9c39-4472-9de2-33435fc76279 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '44e8c9a3ea934c1c96f9c477a8343da6'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:22.525149 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-36366fb9-9c39-4472-9de2-33435fc76279 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '44e8c9a3ea934c1c96f9c477a8343da6'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:22.532939 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-36366fb9-9c39-4472-9de2-33435fc76279 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '44e8c9a3ea934c1c96f9c477a8343da6'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:22.538865 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1425/2832] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:15:22 2022] DELETE /identity/v3/users/44e8c9a3ea934c1c96f9c477a8343da6 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:15:22.987434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78f0c76-4ff0-4c6d-a56e-472836561f81 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:22.989094 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-d78f0c76-4ff0-4c6d-a56e-472836561f81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:23.003091 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78f0c76-4ff0-4c6d-a56e-472836561f81 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:23.066573 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69393001-b719-4407-9441-f7a7eea4313e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:23.067202 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69393001-b719-4407-9441-f7a7eea4313e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:23.067481 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69393001-b719-4407-9441-f7a7eea4313e admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:23.067752 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69393001-b719-4407-9441-f7a7eea4313e admin admin] PATH_INFO: `/v3/projects/dcb3b6b73a0a4509b277b576320e0eee` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:23.068887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69393001-b719-4407-9441-f7a7eea4313e admin admin] RBAC: Authorizing `identity:delete_project(project_id=dcb3b6b73a0a4509b277b576320e0eee)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:23.070964 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69393001-b719-4407-9441-f7a7eea4313e admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:23.201969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-69393001-b719-4407-9441-f7a7eea4313e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dcb3b6b73a0a4509b277b576320e0eee', 'request_id': 'req-69393001-b719-4407-9441-f7a7eea4313e'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:23.281064 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-69393001-b719-4407-9441-f7a7eea4313e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dcb3b6b73a0a4509b277b576320e0eee', 'request_id': 'req-69393001-b719-4407-9441-f7a7eea4313e'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:23.287062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-69393001-b719-4407-9441-f7a7eea4313e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dcb3b6b73a0a4509b277b576320e0eee', 'request_id': 'req-69393001-b719-4407-9441-f7a7eea4313e'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:23.299156 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-69393001-b719-4407-9441-f7a7eea4313e admin admin] The token cache is being invalidate because project dcb3b6b73a0a4509b277b576320e0eee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:15:23.299439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-69393001-b719-4407-9441-f7a7eea4313e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:23.300056 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-69393001-b719-4407-9441-f7a7eea4313e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:23.301524 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1408/2833] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:15:22 2022] DELETE /identity/v3/projects/dcb3b6b73a0a4509b277b576320e0eee => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:15:23.305526 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36366fb9-9c39-4472-9de2-33435fc76279 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:23.306992 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-36366fb9-9c39-4472-9de2-33435fc76279 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:23.320368 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36366fb9-9c39-4472-9de2-33435fc76279 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:23.378484 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c966873-8802-418e-9409-035723071c52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:23.379104 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c966873-8802-418e-9409-035723071c52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:23.379380 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c966873-8802-418e-9409-035723071c52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:23.379648 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c966873-8802-418e-9409-035723071c52 admin admin] PATH_INFO: `/v3/users/452094ea3ec44c31a6b6ce039e3f6edd` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:23.380766 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c966873-8802-418e-9409-035723071c52 admin admin] RBAC: Authorizing `identity:delete_user(user_id=452094ea3ec44c31a6b6ce039e3f6edd)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:23.382820 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c966873-8802-418e-9409-035723071c52 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:23.405518 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:15:23.569245 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-5c966873-8802-418e-9409-035723071c52 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '452094ea3ec44c31a6b6ce039e3f6edd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:23.635391 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-5c966873-8802-418e-9409-035723071c52 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '452094ea3ec44c31a6b6ce039e3f6edd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:23.647916 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-5c966873-8802-418e-9409-035723071c52 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '452094ea3ec44c31a6b6ce039e3f6edd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:23.648548 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-5c966873-8802-418e-9409-035723071c52 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '452094ea3ec44c31a6b6ce039e3f6edd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:23.656440 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-5c966873-8802-418e-9409-035723071c52 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '452094ea3ec44c31a6b6ce039e3f6edd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:23.663121 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1426/2834] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:15:23 2022] DELETE /identity/v3/users/452094ea3ec44c31a6b6ce039e3f6edd => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:15:24.063198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69393001-b719-4407-9441-f7a7eea4313e admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:24.064661 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-69393001-b719-4407-9441-f7a7eea4313e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:24.072415 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:15:24.078595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69393001-b719-4407-9441-f7a7eea4313e admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:24.137846 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95b2e73-cb46-4054-850c-dec3646c6508 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:24.140282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95b2e73-cb46-4054-850c-dec3646c6508 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:24.140566 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95b2e73-cb46-4054-850c-dec3646c6508 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:24.140866 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95b2e73-cb46-4054-850c-dec3646c6508 admin admin] PATH_INFO: `/v3/projects/87b5a1ebba1242c4a1ad1373cf097022` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:24.142175 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95b2e73-cb46-4054-850c-dec3646c6508 admin admin] RBAC: Authorizing `identity:delete_project(project_id=87b5a1ebba1242c4a1ad1373cf097022)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:24.144296 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95b2e73-cb46-4054-850c-dec3646c6508 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:24.260369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a95b2e73-cb46-4054-850c-dec3646c6508 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '87b5a1ebba1242c4a1ad1373cf097022', 'request_id': 'req-a95b2e73-cb46-4054-850c-dec3646c6508'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:24.351115 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a95b2e73-cb46-4054-850c-dec3646c6508 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '87b5a1ebba1242c4a1ad1373cf097022', 'request_id': 'req-a95b2e73-cb46-4054-850c-dec3646c6508'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:24.357051 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a95b2e73-cb46-4054-850c-dec3646c6508 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '87b5a1ebba1242c4a1ad1373cf097022', 'request_id': 'req-a95b2e73-cb46-4054-850c-dec3646c6508'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:24.368230 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a95b2e73-cb46-4054-850c-dec3646c6508 admin admin] The token cache is being invalidate because project 87b5a1ebba1242c4a1ad1373cf097022 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:15:24.368510 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a95b2e73-cb46-4054-850c-dec3646c6508 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:24.369139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-a95b2e73-cb46-4054-850c-dec3646c6508 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:24.370661 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1409/2835] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:15:24 2022] DELETE /identity/v3/projects/87b5a1ebba1242c4a1ad1373cf097022 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:15:24.377173 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47865c86-5c2c-46fe-8fd0-0e0f39523b51 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:24.377691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47865c86-5c2c-46fe-8fd0-0e0f39523b51 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:24.377691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47865c86-5c2c-46fe-8fd0-0e0f39523b51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:24.398068 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-47865c86-5c2c-46fe-8fd0-0e0f39523b51 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:15:24.399295 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-47865c86-5c2c-46fe-8fd0-0e0f39523b51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:24.457179 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1427/2836] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:15:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:15:24.465646 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d09d7f-bc4b-4f8c-a61f-5b98ac21da48 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:24.466067 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d09d7f-bc4b-4f8c-a61f-5b98ac21da48 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:24.466377 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d09d7f-bc4b-4f8c-a61f-5b98ac21da48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:24.487226 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-42d09d7f-bc4b-4f8c-a61f-5b98ac21da48 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:15:24.488581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-42d09d7f-bc4b-4f8c-a61f-5b98ac21da48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:24.496451 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1410/2837] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:15:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:15:24.502991 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47865c86-5c2c-46fe-8fd0-0e0f39523b51 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:24.512970 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47865c86-5c2c-46fe-8fd0-0e0f39523b51 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:24.517662 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b61cd33-f575-4b25-85d6-c3a5a985aee3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:24.518252 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b61cd33-f575-4b25-85d6-c3a5a985aee3 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:24.518570 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b61cd33-f575-4b25-85d6-c3a5a985aee3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:24.518839 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b61cd33-f575-4b25-85d6-c3a5a985aee3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:24.519478 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b61cd33-f575-4b25-85d6-c3a5a985aee3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:15:24.519776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b61cd33-f575-4b25-85d6-c3a5a985aee3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:24.521355 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b61cd33-f575-4b25-85d6-c3a5a985aee3 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:24.532961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1428/2838] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:15:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:15:24.537282 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d09d7f-bc4b-4f8c-a61f-5b98ac21da48 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:24.547280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d09d7f-bc4b-4f8c-a61f-5b98ac21da48 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:24.551939 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff03098-eba1-496e-9b2b-8f8229aae944 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:24.552534 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff03098-eba1-496e-9b2b-8f8229aae944 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:24.552813 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff03098-eba1-496e-9b2b-8f8229aae944 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:24.553085 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff03098-eba1-496e-9b2b-8f8229aae944 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:24.553714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff03098-eba1-496e-9b2b-8f8229aae944 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:24.555632 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff03098-eba1-496e-9b2b-8f8229aae944 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:24.671614 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1411/2839] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:15:24 2022] POST /identity/v3/projects => generated 385 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:15:24.675709 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b61cd33-f575-4b25-85d6-c3a5a985aee3 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:24.686677 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b61cd33-f575-4b25-85d6-c3a5a985aee3 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:24.741996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd40ceab-d8d4-4d9f-9406-0fddab04690e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:24.742697 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd40ceab-d8d4-4d9f-9406-0fddab04690e admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:24.742973 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd40ceab-d8d4-4d9f-9406-0fddab04690e admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:24.743239 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd40ceab-d8d4-4d9f-9406-0fddab04690e admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:24.743860 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd40ceab-d8d4-4d9f-9406-0fddab04690e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:24.745755 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd40ceab-d8d4-4d9f-9406-0fddab04690e admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:24.830872 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1429/2840] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:15:24 2022] POST /identity/v3/users => generated 293 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:15:24.835123 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff03098-eba1-496e-9b2b-8f8229aae944 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:24.845539 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff03098-eba1-496e-9b2b-8f8229aae944 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:24.850134 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f365e59b-7402-4e65-b8e3-8be38ebed893 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:24.851363 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365e59b-7402-4e65-b8e3-8be38ebed893 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:24.851363 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365e59b-7402-4e65-b8e3-8be38ebed893 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:24.851363 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365e59b-7402-4e65-b8e3-8be38ebed893 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:24.852052 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f365e59b-7402-4e65-b8e3-8be38ebed893 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:24.853630 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f365e59b-7402-4e65-b8e3-8be38ebed893 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:24.864242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1412/2841] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:15:24 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:15:24.868283 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd40ceab-d8d4-4d9f-9406-0fddab04690e admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:24.879481 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd40ceab-d8d4-4d9f-9406-0fddab04690e admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:24.884217 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdc5703-bbc7-4a7e-b223-03c64b6cee47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:24.884804 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdc5703-bbc7-4a7e-b223-03c64b6cee47 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:24.885078 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdc5703-bbc7-4a7e-b223-03c64b6cee47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:24.885346 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdc5703-bbc7-4a7e-b223-03c64b6cee47 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:24.885985 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acdc5703-bbc7-4a7e-b223-03c64b6cee47 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:24.887616 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acdc5703-bbc7-4a7e-b223-03c64b6cee47 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:24.898572 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1430/2842] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:15:24 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:15:24.902693 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f365e59b-7402-4e65-b8e3-8be38ebed893 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:24.913346 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f365e59b-7402-4e65-b8e3-8be38ebed893 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:24.918047 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0c3d92-b088-4dda-b45e-4366307f2b32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:24.918710 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0c3d92-b088-4dda-b45e-4366307f2b32 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:24.918994 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0c3d92-b088-4dda-b45e-4366307f2b32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:24.920197 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0c3d92-b088-4dda-b45e-4366307f2b32 admin admin] PATH_INFO: `/v3/projects/e19b3e2dee424dbc9720d0630c5410fe/users/934015e30dcf44188cb26b4a97c75f82/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:24.936449 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0c3d92-b088-4dda-b45e-4366307f2b32 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e19b3e2dee424dbc9720d0630c5410fe, user_id=934015e30dcf44188cb26b4a97c75f82, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:24.939101 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0c3d92-b088-4dda-b45e-4366307f2b32 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:25.013849 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1413/2843] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:15:24 2022] PUT /identity/v3/projects/e19b3e2dee424dbc9720d0630c5410fe/users/934015e30dcf44188cb26b4a97c75f82/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:15:25.078512 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18eb34d-2637-4f9d-826c-deb3714118a9 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:25.078782 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18eb34d-2637-4f9d-826c-deb3714118a9 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:25.079088 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18eb34d-2637-4f9d-826c-deb3714118a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:25.087202 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:15:25.107702 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-d18eb34d-2637-4f9d-826c-deb3714118a9 None None] MFA Rules not processed for user `934015e30dcf44188cb26b4a97c75f82`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:15:25.108996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d18eb34d-2637-4f9d-826c-deb3714118a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:25.175854 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1431/2844] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:15:25 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:15:25.180403 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0c3d92-b088-4dda-b45e-4366307f2b32 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:25.211693 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:15:25.241824 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0c3d92-b088-4dda-b45e-4366307f2b32 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:25.246437 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd89763c-a254-4d10-88bb-37691e0be729 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:25.247031 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd89763c-a254-4d10-88bb-37691e0be729 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:25.247921 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd89763c-a254-4d10-88bb-37691e0be729 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:25.248146 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd89763c-a254-4d10-88bb-37691e0be729 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:25.248979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd89763c-a254-4d10-88bb-37691e0be729 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:25.250877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd89763c-a254-4d10-88bb-37691e0be729 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:25.467690 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1414/2845] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:15:25 2022] POST /identity/v3/projects => generated 385 bytes in 288 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:15:25.471672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18eb34d-2637-4f9d-826c-deb3714118a9 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:25.483195 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18eb34d-2637-4f9d-826c-deb3714118a9 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:25.540355 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938216db-a2b1-4ce8-a17d-603e9b7a1125 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:25.540961 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938216db-a2b1-4ce8-a17d-603e9b7a1125 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:25.541247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938216db-a2b1-4ce8-a17d-603e9b7a1125 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:25.541518 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938216db-a2b1-4ce8-a17d-603e9b7a1125 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:25.542155 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938216db-a2b1-4ce8-a17d-603e9b7a1125 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:25.544825 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938216db-a2b1-4ce8-a17d-603e9b7a1125 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:25.629596 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1432/2846] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:15:25 2022] POST /identity/v3/users => generated 299 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:15:25.633541 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd89763c-a254-4d10-88bb-37691e0be729 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:25.644244 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd89763c-a254-4d10-88bb-37691e0be729 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:25.648852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0326e37-1dc1-4127-8519-73408b53933b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:25.649437 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0326e37-1dc1-4127-8519-73408b53933b admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:25.649712 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0326e37-1dc1-4127-8519-73408b53933b admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:25.649977 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0326e37-1dc1-4127-8519-73408b53933b admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:25.650637 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0326e37-1dc1-4127-8519-73408b53933b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:25.652284 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0326e37-1dc1-4127-8519-73408b53933b admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:25.663014 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1415/2847] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:15:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:15:25.667054 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938216db-a2b1-4ce8-a17d-603e9b7a1125 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:25.677567 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938216db-a2b1-4ce8-a17d-603e9b7a1125 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:25.682138 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edfb9e5-a0af-4ae0-a2fb-07270a340d18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:25.682828 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edfb9e5-a0af-4ae0-a2fb-07270a340d18 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:25.683111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edfb9e5-a0af-4ae0-a2fb-07270a340d18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:25.683383 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6edfb9e5-a0af-4ae0-a2fb-07270a340d18 admin admin] PATH_INFO: `/v3/projects/6c4f6df631dc48d69bb419ce49d0f6bc/users/dc8d9f63bf5b4bfa919d1ccb91f96ebe/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:25.698041 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6edfb9e5-a0af-4ae0-a2fb-07270a340d18 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6c4f6df631dc48d69bb419ce49d0f6bc, user_id=dc8d9f63bf5b4bfa919d1ccb91f96ebe, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:25.701305 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6edfb9e5-a0af-4ae0-a2fb-07270a340d18 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:25.789659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1433/2848] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:15:25 2022] PUT /identity/v3/projects/6c4f6df631dc48d69bb419ce49d0f6bc/users/dc8d9f63bf5b4bfa919d1ccb91f96ebe/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:15:25.793721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0326e37-1dc1-4127-8519-73408b53933b admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:25.804066 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0326e37-1dc1-4127-8519-73408b53933b admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:25.862172 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b13fe9-e210-48da-8351-52da5181e309 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:25.862820 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b13fe9-e210-48da-8351-52da5181e309 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:25.863098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b13fe9-e210-48da-8351-52da5181e309 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:25.863369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b13fe9-e210-48da-8351-52da5181e309 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:25.864001 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b13fe9-e210-48da-8351-52da5181e309 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:25.865581 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b13fe9-e210-48da-8351-52da5181e309 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:25.875242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1416/2849] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:15:25 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:15:25.879465 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edfb9e5-a0af-4ae0-a2fb-07270a340d18 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:25.890353 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6edfb9e5-a0af-4ae0-a2fb-07270a340d18 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:25.894886 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b1306e-e1e6-4b5a-a2df-e154d859c338 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:25.895521 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b1306e-e1e6-4b5a-a2df-e154d859c338 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:25.895794 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b1306e-e1e6-4b5a-a2df-e154d859c338 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:25.896061 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b1306e-e1e6-4b5a-a2df-e154d859c338 admin admin] PATH_INFO: `/v3/projects/6c4f6df631dc48d69bb419ce49d0f6bc/users/dc8d9f63bf5b4bfa919d1ccb91f96ebe/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:25.897936 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b1306e-e1e6-4b5a-a2df-e154d859c338 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6c4f6df631dc48d69bb419ce49d0f6bc, user_id=dc8d9f63bf5b4bfa919d1ccb91f96ebe, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:25.900618 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b1306e-e1e6-4b5a-a2df-e154d859c338 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:25.911278 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1434/2850] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:15:25 2022] PUT /identity/v3/projects/6c4f6df631dc48d69bb419ce49d0f6bc/users/dc8d9f63bf5b4bfa919d1ccb91f96ebe/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:15:25.921536 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e205392e-52d3-4916-82a9-9a191177e227 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:25.921818 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e205392e-52d3-4916-82a9-9a191177e227 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:25.922088 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e205392e-52d3-4916-82a9-9a191177e227 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:25.948862 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-e205392e-52d3-4916-82a9-9a191177e227 None None] MFA Rules not processed for user `dc8d9f63bf5b4bfa919d1ccb91f96ebe`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:15:25.950052 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e205392e-52d3-4916-82a9-9a191177e227 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:26.017383 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1417/2851] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:15:25 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:15:26.028745 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b1306e-e1e6-4b5a-a2df-e154d859c338 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:26.030476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d4b1306e-e1e6-4b5a-a2df-e154d859c338 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:26.083066 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:15:26.088158 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b1306e-e1e6-4b5a-a2df-e154d859c338 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:26.091833 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cbc5e1-5b56-4c3e-93d3-7d3ad6019bf9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:26.092433 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cbc5e1-5b56-4c3e-93d3-7d3ad6019bf9 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:26.092712 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cbc5e1-5b56-4c3e-93d3-7d3ad6019bf9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:26.092978 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cbc5e1-5b56-4c3e-93d3-7d3ad6019bf9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:26.150408 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11cbc5e1-5b56-4c3e-93d3-7d3ad6019bf9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:26.152180 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11cbc5e1-5b56-4c3e-93d3-7d3ad6019bf9 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:26.162125 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1435/2852] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:15:26 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:15:31.339021 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e205392e-52d3-4916-82a9-9a191177e227 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:31.340439 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-e205392e-52d3-4916-82a9-9a191177e227 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:31.352434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e205392e-52d3-4916-82a9-9a191177e227 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:31.356001 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c1d903-a3c3-43d8-8377-be214ade3c15 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:31.356595 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c1d903-a3c3-43d8-8377-be214ade3c15 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:31.356877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c1d903-a3c3-43d8-8377-be214ade3c15 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:31.357145 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c1d903-a3c3-43d8-8377-be214ade3c15 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:31.366037 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c1d903-a3c3-43d8-8377-be214ade3c15 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:31.367482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c1d903-a3c3-43d8-8377-be214ade3c15 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:31.377337 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1418/2853] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:15:31 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:15:36.157463 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=27 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:15:37.412226 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cbc5e1-5b56-4c3e-93d3-7d3ad6019bf9 service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:37.473303 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cbc5e1-5b56-4c3e-93d3-7d3ad6019bf9 service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:37.477730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbbf5e7-bf42-4c20-ac71-fb3649614f70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:37.478353 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbbf5e7-bf42-4c20-ac71-fb3649614f70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:37.478645 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbbf5e7-bf42-4c20-ac71-fb3649614f70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:37.478910 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdbbf5e7-bf42-4c20-ac71-fb3649614f70 admin admin] PATH_INFO: `/v3/users/934015e30dcf44188cb26b4a97c75f82` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:37.480009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdbbf5e7-bf42-4c20-ac71-fb3649614f70 admin admin] RBAC: Authorizing `identity:delete_user(user_id=934015e30dcf44188cb26b4a97c75f82)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:37.481907 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdbbf5e7-bf42-4c20-ac71-fb3649614f70 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:37.677886 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-cdbbf5e7-bf42-4c20-ac71-fb3649614f70 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '934015e30dcf44188cb26b4a97c75f82'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:37.715812 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-cdbbf5e7-bf42-4c20-ac71-fb3649614f70 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '934015e30dcf44188cb26b4a97c75f82'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:37.729040 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-cdbbf5e7-bf42-4c20-ac71-fb3649614f70 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '934015e30dcf44188cb26b4a97c75f82'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:37.729593 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-cdbbf5e7-bf42-4c20-ac71-fb3649614f70 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '934015e30dcf44188cb26b4a97c75f82'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:37.736735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-cdbbf5e7-bf42-4c20-ac71-fb3649614f70 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '934015e30dcf44188cb26b4a97c75f82'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:37.742665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1436/2854] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:15:37 2022] DELETE /identity/v3/users/934015e30dcf44188cb26b4a97c75f82 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:15:37.747405 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57593db-8d23-4f75-9457-6f26dc0c1515 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:37.747899 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57593db-8d23-4f75-9457-6f26dc0c1515 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:37.748006 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57593db-8d23-4f75-9457-6f26dc0c1515 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:37.767716 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-c57593db-8d23-4f75-9457-6f26dc0c1515 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:15:37.768951 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-c57593db-8d23-4f75-9457-6f26dc0c1515 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:37.773855 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:15:37.826496 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1419/2855] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:15:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:15:37.836888 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbbf5e7-bf42-4c20-ac71-fb3649614f70 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:37.838301 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-cdbbf5e7-bf42-4c20-ac71-fb3649614f70 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:37.895993 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdbbf5e7-bf42-4c20-ac71-fb3649614f70 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:37.899701 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0842d500-9919-4c5e-a1c3-d885e330b32e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:37.900301 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0842d500-9919-4c5e-a1c3-d885e330b32e service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:37.900576 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0842d500-9919-4c5e-a1c3-d885e330b32e service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:37.900843 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0842d500-9919-4c5e-a1c3-d885e330b32e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:37.910062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0842d500-9919-4c5e-a1c3-d885e330b32e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:37.911523 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0842d500-9919-4c5e-a1c3-d885e330b32e service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:37.921074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1437/2856] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:15:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:15:38.338837 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57593db-8d23-4f75-9457-6f26dc0c1515 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:38.340347 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-c57593db-8d23-4f75-9457-6f26dc0c1515 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:38.359207 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57593db-8d23-4f75-9457-6f26dc0c1515 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:38.366453 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666e8699-cd16-4d65-b868-fab0db31767f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:38.367099 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666e8699-cd16-4d65-b868-fab0db31767f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:38.367346 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666e8699-cd16-4d65-b868-fab0db31767f admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:38.367627 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666e8699-cd16-4d65-b868-fab0db31767f admin admin] PATH_INFO: `/v3/projects/e19b3e2dee424dbc9720d0630c5410fe` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:38.368795 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666e8699-cd16-4d65-b868-fab0db31767f admin admin] RBAC: Authorizing `identity:delete_project(project_id=e19b3e2dee424dbc9720d0630c5410fe)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:38.370755 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666e8699-cd16-4d65-b868-fab0db31767f admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:38.521196 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-666e8699-cd16-4d65-b868-fab0db31767f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e19b3e2dee424dbc9720d0630c5410fe', 'request_id': 'req-666e8699-cd16-4d65-b868-fab0db31767f'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:38.602158 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-666e8699-cd16-4d65-b868-fab0db31767f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e19b3e2dee424dbc9720d0630c5410fe', 'request_id': 'req-666e8699-cd16-4d65-b868-fab0db31767f'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:38.608218 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-666e8699-cd16-4d65-b868-fab0db31767f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e19b3e2dee424dbc9720d0630c5410fe', 'request_id': 'req-666e8699-cd16-4d65-b868-fab0db31767f'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:38.620050 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-666e8699-cd16-4d65-b868-fab0db31767f admin admin] The token cache is being invalidate because project e19b3e2dee424dbc9720d0630c5410fe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:15:38.620336 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-666e8699-cd16-4d65-b868-fab0db31767f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:38.621537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-666e8699-cd16-4d65-b868-fab0db31767f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:38.622636 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1420/2857] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:15:38 2022] DELETE /identity/v3/projects/e19b3e2dee424dbc9720d0630c5410fe => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:15:38.627232 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0842d500-9919-4c5e-a1c3-d885e330b32e service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:38.637308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0842d500-9919-4c5e-a1c3-d885e330b32e service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:38.655288 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:15:38.694685 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd98454-02a0-41ac-ac89-f1f52fc24f56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:38.695314 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd98454-02a0-41ac-ac89-f1f52fc24f56 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:38.695591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd98454-02a0-41ac-ac89-f1f52fc24f56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:38.695868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd98454-02a0-41ac-ac89-f1f52fc24f56 admin admin] PATH_INFO: `/v3/users/dc8d9f63bf5b4bfa919d1ccb91f96ebe` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:38.696989 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd98454-02a0-41ac-ac89-f1f52fc24f56 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dc8d9f63bf5b4bfa919d1ccb91f96ebe)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:38.699035 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd98454-02a0-41ac-ac89-f1f52fc24f56 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:38.913514 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8dd98454-02a0-41ac-ac89-f1f52fc24f56 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dc8d9f63bf5b4bfa919d1ccb91f96ebe'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:39.017997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8dd98454-02a0-41ac-ac89-f1f52fc24f56 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dc8d9f63bf5b4bfa919d1ccb91f96ebe'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:39.031184 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8dd98454-02a0-41ac-ac89-f1f52fc24f56 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dc8d9f63bf5b4bfa919d1ccb91f96ebe'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:39.038229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8dd98454-02a0-41ac-ac89-f1f52fc24f56 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dc8d9f63bf5b4bfa919d1ccb91f96ebe'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:39.039274 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-8dd98454-02a0-41ac-ac89-f1f52fc24f56 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dc8d9f63bf5b4bfa919d1ccb91f96ebe'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:39.045695 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1438/2858] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:15:38 2022] DELETE /identity/v3/users/dc8d9f63bf5b4bfa919d1ccb91f96ebe => generated 0 bytes in 420 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:15:39.569960 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666e8699-cd16-4d65-b868-fab0db31767f admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:39.571506 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-666e8699-cd16-4d65-b868-fab0db31767f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:39.586215 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666e8699-cd16-4d65-b868-fab0db31767f admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:39.645482 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc7d7bd-76e7-4e07-ab02-ef32ac164cf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:39.646119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc7d7bd-76e7-4e07-ab02-ef32ac164cf9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:39.646476 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc7d7bd-76e7-4e07-ab02-ef32ac164cf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:39.646692 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc7d7bd-76e7-4e07-ab02-ef32ac164cf9 admin admin] PATH_INFO: `/v3/projects/6c4f6df631dc48d69bb419ce49d0f6bc` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:39.647834 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc7d7bd-76e7-4e07-ab02-ef32ac164cf9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6c4f6df631dc48d69bb419ce49d0f6bc)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:39.649759 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc7d7bd-76e7-4e07-ab02-ef32ac164cf9 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:39.778933 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:15:39.789334 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3bc7d7bd-76e7-4e07-ab02-ef32ac164cf9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6c4f6df631dc48d69bb419ce49d0f6bc', 'request_id': 'req-3bc7d7bd-76e7-4e07-ab02-ef32ac164cf9'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:39.897571 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3bc7d7bd-76e7-4e07-ab02-ef32ac164cf9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6c4f6df631dc48d69bb419ce49d0f6bc', 'request_id': 'req-3bc7d7bd-76e7-4e07-ab02-ef32ac164cf9'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:39.903773 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3bc7d7bd-76e7-4e07-ab02-ef32ac164cf9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6c4f6df631dc48d69bb419ce49d0f6bc', 'request_id': 'req-3bc7d7bd-76e7-4e07-ab02-ef32ac164cf9'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:39.916015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3bc7d7bd-76e7-4e07-ab02-ef32ac164cf9 admin admin] The token cache is being invalidate because project 6c4f6df631dc48d69bb419ce49d0f6bc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:15:39.916301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3bc7d7bd-76e7-4e07-ab02-ef32ac164cf9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:39.917007 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3bc7d7bd-76e7-4e07-ab02-ef32ac164cf9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:15:39.918521 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1421/2859] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:15:39 2022] DELETE /identity/v3/projects/6c4f6df631dc48d69bb419ce49d0f6bc => generated 0 bytes in 349 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:15:39.925107 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1248aa13-c56f-4dee-b1b8-2c95a93394a5 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:39.925620 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1248aa13-c56f-4dee-b1b8-2c95a93394a5 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:39.925728 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1248aa13-c56f-4dee-b1b8-2c95a93394a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:39.946733 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-1248aa13-c56f-4dee-b1b8-2c95a93394a5 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:15:39.947971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-1248aa13-c56f-4dee-b1b8-2c95a93394a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:40.008826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1439/2860] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:15:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:15:40.018385 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cef4087-e532-4b80-99fb-3a23ca93a5d6 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:40.018665 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cef4087-e532-4b80-99fb-3a23ca93a5d6 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:40.018930 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cef4087-e532-4b80-99fb-3a23ca93a5d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:40.040184 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-5cef4087-e532-4b80-99fb-3a23ca93a5d6 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:15:40.041459 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-5cef4087-e532-4b80-99fb-3a23ca93a5d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:40.050527 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1422/2861] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:15:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:15:40.055056 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1248aa13-c56f-4dee-b1b8-2c95a93394a5 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:40.065152 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1248aa13-c56f-4dee-b1b8-2c95a93394a5 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:40.069840 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbd5d8f-f80f-40ca-9311-1f4971008579 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:40.070465 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbd5d8f-f80f-40ca-9311-1f4971008579 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:40.070746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbd5d8f-f80f-40ca-9311-1f4971008579 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:40.071013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bbd5d8f-f80f-40ca-9311-1f4971008579 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:40.071657 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbd5d8f-f80f-40ca-9311-1f4971008579 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:15:40.071949 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbd5d8f-f80f-40ca-9311-1f4971008579 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:40.073591 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bbd5d8f-f80f-40ca-9311-1f4971008579 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:40.086786 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1440/2862] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:15:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:15:40.091483 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cef4087-e532-4b80-99fb-3a23ca93a5d6 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:40.102443 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cef4087-e532-4b80-99fb-3a23ca93a5d6 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:40.107485 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3f99be-26a5-4bf1-94d3-e98a562f247d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:40.108111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3f99be-26a5-4bf1-94d3-e98a562f247d admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:40.108394 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3f99be-26a5-4bf1-94d3-e98a562f247d admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:40.108668 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3f99be-26a5-4bf1-94d3-e98a562f247d admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:40.109322 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3f99be-26a5-4bf1-94d3-e98a562f247d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:40.111666 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3f99be-26a5-4bf1-94d3-e98a562f247d admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:40.216137 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1423/2863] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:15:40 2022] POST /identity/v3/projects => generated 377 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:15:40.221753 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbd5d8f-f80f-40ca-9311-1f4971008579 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:40.231954 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bbd5d8f-f80f-40ca-9311-1f4971008579 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:40.258505 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:15:40.290260 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6a056e-17c2-4b76-afc5-b3b044523014 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:40.290912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6a056e-17c2-4b76-afc5-b3b044523014 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:40.291207 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6a056e-17c2-4b76-afc5-b3b044523014 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:40.291441 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6a056e-17c2-4b76-afc5-b3b044523014 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:40.292214 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6a056e-17c2-4b76-afc5-b3b044523014 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:40.293971 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6a056e-17c2-4b76-afc5-b3b044523014 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:40.366102 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1441/2864] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:15:40 2022] POST /identity/v3/users => generated 289 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:15:40.370380 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3f99be-26a5-4bf1-94d3-e98a562f247d admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:40.381616 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3f99be-26a5-4bf1-94d3-e98a562f247d admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:40.386525 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc2c963-b4af-4f8b-9965-32a7edd7a26f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:40.387129 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc2c963-b4af-4f8b-9965-32a7edd7a26f admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:40.387409 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc2c963-b4af-4f8b-9965-32a7edd7a26f admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:40.387681 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffc2c963-b4af-4f8b-9965-32a7edd7a26f admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:40.388325 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc2c963-b4af-4f8b-9965-32a7edd7a26f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:40.389919 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffc2c963-b4af-4f8b-9965-32a7edd7a26f admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:40.400929 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1424/2865] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:15:40 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:15:40.404951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6a056e-17c2-4b76-afc5-b3b044523014 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:40.415672 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6a056e-17c2-4b76-afc5-b3b044523014 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:40.420758 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba45010-638e-46b7-b36a-8afcfdbaa30f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:40.421361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba45010-638e-46b7-b36a-8afcfdbaa30f admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:40.421665 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba45010-638e-46b7-b36a-8afcfdbaa30f admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:40.421937 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba45010-638e-46b7-b36a-8afcfdbaa30f admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:40.422625 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba45010-638e-46b7-b36a-8afcfdbaa30f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:40.424257 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba45010-638e-46b7-b36a-8afcfdbaa30f admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:40.434655 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1442/2866] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:15:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:15:40.439474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc2c963-b4af-4f8b-9965-32a7edd7a26f admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:40.449825 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffc2c963-b4af-4f8b-9965-32a7edd7a26f admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:40.455192 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7297a33a-8abf-48be-bf99-40bab5c71b66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:40.455877 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7297a33a-8abf-48be-bf99-40bab5c71b66 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:40.456157 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7297a33a-8abf-48be-bf99-40bab5c71b66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:40.456434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7297a33a-8abf-48be-bf99-40bab5c71b66 admin admin] PATH_INFO: `/v3/projects/e2b4fe90f9714a55be65b95a95713e2e/users/43bf40312af24d9f8535f9ebb45ef72f/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:40.471075 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7297a33a-8abf-48be-bf99-40bab5c71b66 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2b4fe90f9714a55be65b95a95713e2e, user_id=43bf40312af24d9f8535f9ebb45ef72f, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:40.473722 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7297a33a-8abf-48be-bf99-40bab5c71b66 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:40.573587 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1425/2867] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:15:40 2022] PUT /identity/v3/projects/e2b4fe90f9714a55be65b95a95713e2e/users/43bf40312af24d9f8535f9ebb45ef72f/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:15:40.584834 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b0138a-0844-44b3-bb76-c0d7a2e15d4c None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:40.585227 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b0138a-0844-44b3-bb76-c0d7a2e15d4c None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:40.585602 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b0138a-0844-44b3-bb76-c0d7a2e15d4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:40.614954 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-38b0138a-0844-44b3-bb76-c0d7a2e15d4c None None] MFA Rules not processed for user `43bf40312af24d9f8535f9ebb45ef72f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:15:40.614954 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-38b0138a-0844-44b3-bb76-c0d7a2e15d4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:40.682103 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1443/2868] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:15:40 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:15:40.694913 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7297a33a-8abf-48be-bf99-40bab5c71b66 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:15:40.696638 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-7297a33a-8abf-48be-bf99-40bab5c71b66 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:15:40.753101 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7297a33a-8abf-48be-bf99-40bab5c71b66 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:15:40.756934 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1b66f3-7b5e-440a-a639-f1a4cb508fe2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:15:40.757541 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1b66f3-7b5e-440a-a639-f1a4cb508fe2 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:15:40.757822 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1b66f3-7b5e-440a-a639-f1a4cb508fe2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:15:40.758096 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1b66f3-7b5e-440a-a639-f1a4cb508fe2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:15:40.766708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1b66f3-7b5e-440a-a639-f1a4cb508fe2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:15:40.768284 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1b66f3-7b5e-440a-a639-f1a4cb508fe2 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:15:40.777011 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1426/2869] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:15:40 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:15:50.678977 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=53,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:15:50.772979 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=64,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:16:06.135439 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b0138a-0844-44b3-bb76-c0d7a2e15d4c None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:06.199097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b0138a-0844-44b3-bb76-c0d7a2e15d4c None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:06.204629 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f75c9b4-1bd7-4d79-b145-bb798fa5542d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:06.205268 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f75c9b4-1bd7-4d79-b145-bb798fa5542d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:06.205546 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f75c9b4-1bd7-4d79-b145-bb798fa5542d admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:06.205848 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f75c9b4-1bd7-4d79-b145-bb798fa5542d admin admin] PATH_INFO: `/v3/users/43bf40312af24d9f8535f9ebb45ef72f` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:06.207554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f75c9b4-1bd7-4d79-b145-bb798fa5542d admin admin] RBAC: Authorizing `identity:delete_user(user_id=43bf40312af24d9f8535f9ebb45ef72f)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:06.209548 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f75c9b4-1bd7-4d79-b145-bb798fa5542d admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:06.406519 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0f75c9b4-1bd7-4d79-b145-bb798fa5542d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '43bf40312af24d9f8535f9ebb45ef72f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:16:06.472819 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0f75c9b4-1bd7-4d79-b145-bb798fa5542d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '43bf40312af24d9f8535f9ebb45ef72f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:16:06.485263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0f75c9b4-1bd7-4d79-b145-bb798fa5542d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '43bf40312af24d9f8535f9ebb45ef72f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:16:06.485894 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0f75c9b4-1bd7-4d79-b145-bb798fa5542d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '43bf40312af24d9f8535f9ebb45ef72f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:16:06.493046 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-0f75c9b4-1bd7-4d79-b145-bb798fa5542d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '43bf40312af24d9f8535f9ebb45ef72f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:16:06.498927 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1444/2870] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:16:06 2022] DELETE /identity/v3/users/43bf40312af24d9f8535f9ebb45ef72f => generated 0 bytes in 364 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:16:06.503483 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4a15c5-ac32-487a-ba26-6b7e453c6add None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:06.504544 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4a15c5-ac32-487a-ba26-6b7e453c6add None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:06.504635 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e4a15c5-ac32-487a-ba26-6b7e453c6add None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:06.524927 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-0e4a15c5-ac32-487a-ba26-6b7e453c6add None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:16:06.526125 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-0e4a15c5-ac32-487a-ba26-6b7e453c6add None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:06.583019 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1427/2871] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:16:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:16:06.592541 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f75c9b4-1bd7-4d79-b145-bb798fa5542d admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:06.593919 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-0f75c9b4-1bd7-4d79-b145-bb798fa5542d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:06.650660 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f75c9b4-1bd7-4d79-b145-bb798fa5542d admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:06.654228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c90f374-df7b-482b-85b0-e3fc04e5e7fa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:06.654846 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c90f374-df7b-482b-85b0-e3fc04e5e7fa service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:06.655120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c90f374-df7b-482b-85b0-e3fc04e5e7fa service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:06.655387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c90f374-df7b-482b-85b0-e3fc04e5e7fa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:06.664217 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c90f374-df7b-482b-85b0-e3fc04e5e7fa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:06.665549 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c90f374-df7b-482b-85b0-e3fc04e5e7fa service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:06.675188 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1445/2872] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:16:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:16:07.082991 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4a15c5-ac32-487a-ba26-6b7e453c6add None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:07.084525 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-0e4a15c5-ac32-487a-ba26-6b7e453c6add None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:07.097851 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e4a15c5-ac32-487a-ba26-6b7e453c6add None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:07.104852 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902395cf-04ee-4716-aa69-5f9ba4dfe07f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:07.105715 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902395cf-04ee-4716-aa69-5f9ba4dfe07f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:07.106951 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902395cf-04ee-4716-aa69-5f9ba4dfe07f admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:07.107559 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902395cf-04ee-4716-aa69-5f9ba4dfe07f admin admin] PATH_INFO: `/v3/projects/e2b4fe90f9714a55be65b95a95713e2e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:07.108726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902395cf-04ee-4716-aa69-5f9ba4dfe07f admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2b4fe90f9714a55be65b95a95713e2e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:07.110720 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902395cf-04ee-4716-aa69-5f9ba4dfe07f admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:07.233220 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-902395cf-04ee-4716-aa69-5f9ba4dfe07f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2b4fe90f9714a55be65b95a95713e2e', 'request_id': 'req-902395cf-04ee-4716-aa69-5f9ba4dfe07f'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:16:07.297802 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-902395cf-04ee-4716-aa69-5f9ba4dfe07f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2b4fe90f9714a55be65b95a95713e2e', 'request_id': 'req-902395cf-04ee-4716-aa69-5f9ba4dfe07f'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:16:07.304431 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-902395cf-04ee-4716-aa69-5f9ba4dfe07f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2b4fe90f9714a55be65b95a95713e2e', 'request_id': 'req-902395cf-04ee-4716-aa69-5f9ba4dfe07f'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:16:07.315479 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-902395cf-04ee-4716-aa69-5f9ba4dfe07f admin admin] The token cache is being invalidate because project e2b4fe90f9714a55be65b95a95713e2e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81569) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:16:07.315767 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-902395cf-04ee-4716-aa69-5f9ba4dfe07f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:16:07.316621 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-902395cf-04ee-4716-aa69-5f9ba4dfe07f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:16:07.318062 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1428/2873] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:16:07 2022] DELETE /identity/v3/projects/e2b4fe90f9714a55be65b95a95713e2e => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:16:07.323009 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961269ee-f8fc-4ccb-8e2b-9a23ca4f3b0a None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:07.323501 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961269ee-f8fc-4ccb-8e2b-9a23ca4f3b0a None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:07.323606 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961269ee-f8fc-4ccb-8e2b-9a23ca4f3b0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:07.342900 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-961269ee-f8fc-4ccb-8e2b-9a23ca4f3b0a None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:16:07.344098 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-961269ee-f8fc-4ccb-8e2b-9a23ca4f3b0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:07.361605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:16:07.404243 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1446/2874] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:16:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:16:07.412658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f85230-a5a6-41e7-a440-bdffb11274ff None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:07.412944 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f85230-a5a6-41e7-a440-bdffb11274ff None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:07.413220 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f85230-a5a6-41e7-a440-bdffb11274ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:07.433448 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-68f85230-a5a6-41e7-a440-bdffb11274ff None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:16:07.434735 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-68f85230-a5a6-41e7-a440-bdffb11274ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:07.443156 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1429/2875] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:16:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:16:07.447517 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961269ee-f8fc-4ccb-8e2b-9a23ca4f3b0a None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:07.458399 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961269ee-f8fc-4ccb-8e2b-9a23ca4f3b0a None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:07.462891 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a0e8c6-aa5d-411c-90ec-ceac1a0b7135 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:07.463476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a0e8c6-aa5d-411c-90ec-ceac1a0b7135 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:07.463752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a0e8c6-aa5d-411c-90ec-ceac1a0b7135 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:07.464020 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a0e8c6-aa5d-411c-90ec-ceac1a0b7135 admin admin] PATH_INFO: `/v3/domains` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:07.464661 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a0e8c6-aa5d-411c-90ec-ceac1a0b7135 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81568) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:16:07.464955 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a0e8c6-aa5d-411c-90ec-ceac1a0b7135 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:07.466611 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a0e8c6-aa5d-411c-90ec-ceac1a0b7135 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:07.478644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1447/2876] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:16:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:16:07.483019 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f85230-a5a6-41e7-a440-bdffb11274ff None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:07.492330 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f85230-a5a6-41e7-a440-bdffb11274ff None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:07.497176 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de635074-da84-465b-8232-8b39be6b4c26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:07.497787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de635074-da84-465b-8232-8b39be6b4c26 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:07.498070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de635074-da84-465b-8232-8b39be6b4c26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:07.498365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de635074-da84-465b-8232-8b39be6b4c26 admin admin] PATH_INFO: `/v3/projects` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:07.499041 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de635074-da84-465b-8232-8b39be6b4c26 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:07.500929 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de635074-da84-465b-8232-8b39be6b4c26 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:07.579267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1430/2877] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:16:07 2022] POST /identity/v3/projects => generated 371 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:16:07.583914 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a0e8c6-aa5d-411c-90ec-ceac1a0b7135 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:07.594002 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a0e8c6-aa5d-411c-90ec-ceac1a0b7135 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:07.650805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c253b5e-90d9-408d-bc6d-8d4f96aeb479 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:07.651402 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c253b5e-90d9-408d-bc6d-8d4f96aeb479 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:07.651680 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c253b5e-90d9-408d-bc6d-8d4f96aeb479 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:07.651946 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c253b5e-90d9-408d-bc6d-8d4f96aeb479 admin admin] PATH_INFO: `/v3/users` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:07.652568 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c253b5e-90d9-408d-bc6d-8d4f96aeb479 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:07.654487 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c253b5e-90d9-408d-bc6d-8d4f96aeb479 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:07.731610 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1448/2878] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:16:07 2022] POST /identity/v3/users => generated 286 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:16:07.735624 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de635074-da84-465b-8232-8b39be6b4c26 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:07.745947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de635074-da84-465b-8232-8b39be6b4c26 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:07.750535 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0d1875-64cb-4f55-a4ed-955320208a53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:07.751121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0d1875-64cb-4f55-a4ed-955320208a53 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:07.751399 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0d1875-64cb-4f55-a4ed-955320208a53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:07.751666 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0d1875-64cb-4f55-a4ed-955320208a53 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:07.752299 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0d1875-64cb-4f55-a4ed-955320208a53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:07.754139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0d1875-64cb-4f55-a4ed-955320208a53 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:07.763937 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1431/2879] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:16:07 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:16:07.767776 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c253b5e-90d9-408d-bc6d-8d4f96aeb479 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:07.778308 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c253b5e-90d9-408d-bc6d-8d4f96aeb479 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:07.783046 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f0c54e-3b4d-44d6-927d-4b9e3678da03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:07.783631 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f0c54e-3b4d-44d6-927d-4b9e3678da03 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:07.783911 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f0c54e-3b4d-44d6-927d-4b9e3678da03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:07.784181 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f0c54e-3b4d-44d6-927d-4b9e3678da03 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:07.785438 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f0c54e-3b4d-44d6-927d-4b9e3678da03 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:07.787228 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f0c54e-3b4d-44d6-927d-4b9e3678da03 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:07.798028 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1449/2880] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:16:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:16:07.802286 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0d1875-64cb-4f55-a4ed-955320208a53 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:07.812208 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0d1875-64cb-4f55-a4ed-955320208a53 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:07.817737 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca189ee1-191d-4e10-b2ea-5e6cf2a369bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:07.818423 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca189ee1-191d-4e10-b2ea-5e6cf2a369bd admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:07.818708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca189ee1-191d-4e10-b2ea-5e6cf2a369bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:07.818981 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca189ee1-191d-4e10-b2ea-5e6cf2a369bd admin admin] PATH_INFO: `/v3/projects/87b469c9013e44e4af6452e516cf97e2/users/f6ee14289b9e491e9e0223c414ac58ab/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:07.834189 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca189ee1-191d-4e10-b2ea-5e6cf2a369bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=87b469c9013e44e4af6452e516cf97e2, user_id=f6ee14289b9e491e9e0223c414ac58ab, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:07.836867 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca189ee1-191d-4e10-b2ea-5e6cf2a369bd admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:07.914633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1432/2881] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:16:07 2022] PUT /identity/v3/projects/87b469c9013e44e4af6452e516cf97e2/users/f6ee14289b9e491e9e0223c414ac58ab/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:16:07.926480 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a334f9f6-c652-4690-971c-2a0e202a6adb None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:07.929826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a334f9f6-c652-4690-971c-2a0e202a6adb None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:07.929826 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a334f9f6-c652-4690-971c-2a0e202a6adb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:07.955586 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-a334f9f6-c652-4690-971c-2a0e202a6adb None None] MFA Rules not processed for user `f6ee14289b9e491e9e0223c414ac58ab`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:16:07.957454 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a334f9f6-c652-4690-971c-2a0e202a6adb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:08.020293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1450/2882] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:16:07 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:16:08.032243 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca189ee1-191d-4e10-b2ea-5e6cf2a369bd admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:08.033918 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ca189ee1-191d-4e10-b2ea-5e6cf2a369bd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:08.056918 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:16:08.111827 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca189ee1-191d-4e10-b2ea-5e6cf2a369bd admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:08.116097 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186cb957-c8d8-420c-9bd0-9ec64be752f9 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'acdf4a7bd608492390176dcc42ca1267', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:08.116721 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186cb957-c8d8-420c-9bd0-9ec64be752f9 service glance] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:08.116995 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186cb957-c8d8-420c-9bd0-9ec64be752f9 service glance] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:08.117263 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186cb957-c8d8-420c-9bd0-9ec64be752f9 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:08.125668 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186cb957-c8d8-420c-9bd0-9ec64be752f9 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:08.127349 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186cb957-c8d8-420c-9bd0-9ec64be752f9 service glance] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:08.136879 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1433/2883] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:16:08 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:16:18.016749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=89,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:16:18.166289 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=26 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:16:31.535100 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a334f9f6-c652-4690-971c-2a0e202a6adb None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:31.536607 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a334f9f6-c652-4690-971c-2a0e202a6adb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:31.605632 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a334f9f6-c652-4690-971c-2a0e202a6adb None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:31.610763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d2bc9bd-0f48-4515-ab7c-09d3fc5aa4fd service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce249d4a99c34891b6ba941f57a14dd4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:31.611361 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2bc9bd-0f48-4515-ab7c-09d3fc5aa4fd service placement] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:31.611638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2bc9bd-0f48-4515-ab7c-09d3fc5aa4fd service placement] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:31.611911 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d2bc9bd-0f48-4515-ab7c-09d3fc5aa4fd service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:31.613629 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-1d2bc9bd-0f48-4515-ab7c-09d3fc5aa4fd service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:31.625850 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2bc9bd-0f48-4515-ab7c-09d3fc5aa4fd service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:31.627455 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d2bc9bd-0f48-4515-ab7c-09d3fc5aa4fd service placement] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:31.638073 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1451/2884] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:16:31 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:16:37.638764 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186cb957-c8d8-420c-9bd0-9ec64be752f9 service glance] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:37.699940 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186cb957-c8d8-420c-9bd0-9ec64be752f9 service glance] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:37.704440 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f01c61-bb1d-45a6-a73b-06c7b022d0db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:37.705056 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f01c61-bb1d-45a6-a73b-06c7b022d0db admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:37.705334 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f01c61-bb1d-45a6-a73b-06c7b022d0db admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:37.705605 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f01c61-bb1d-45a6-a73b-06c7b022d0db admin admin] PATH_INFO: `/v3/users/f6ee14289b9e491e9e0223c414ac58ab` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:37.706750 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f01c61-bb1d-45a6-a73b-06c7b022d0db admin admin] RBAC: Authorizing `identity:delete_user(user_id=f6ee14289b9e491e9e0223c414ac58ab)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:37.709253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f01c61-bb1d-45a6-a73b-06c7b022d0db admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:37.886989 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-53f01c61-bb1d-45a6-a73b-06c7b022d0db admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f6ee14289b9e491e9e0223c414ac58ab'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:16:37.959232 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-53f01c61-bb1d-45a6-a73b-06c7b022d0db admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f6ee14289b9e491e9e0223c414ac58ab'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:16:37.960401 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-53f01c61-bb1d-45a6-a73b-06c7b022d0db admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f6ee14289b9e491e9e0223c414ac58ab'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:16:37.972329 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-53f01c61-bb1d-45a6-a73b-06c7b022d0db admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f6ee14289b9e491e9e0223c414ac58ab'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:16:37.979108 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-53f01c61-bb1d-45a6-a73b-06c7b022d0db admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f6ee14289b9e491e9e0223c414ac58ab'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:16:37.985047 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1434/2885] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:16:37 2022] DELETE /identity/v3/users/f6ee14289b9e491e9e0223c414ac58ab => generated 0 bytes in 347 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:16:37.989600 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5765d6b6-0f97-4694-925b-e93abf444d8c None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:37.990112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5765d6b6-0f97-4694-925b-e93abf444d8c None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:37.990220 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5765d6b6-0f97-4694-925b-e93abf444d8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:38.009465 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-5765d6b6-0f97-4694-925b-e93abf444d8c None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:16:38.010706 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-5765d6b6-0f97-4694-925b-e93abf444d8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:38.070544 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1452/2886] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:16:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:16:38.080083 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f01c61-bb1d-45a6-a73b-06c7b022d0db admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:38.081462 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-53f01c61-bb1d-45a6-a73b-06c7b022d0db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:38.139668 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f01c61-bb1d-45a6-a73b-06c7b022d0db admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:38.143209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021242c1-a070-4870-a65e-7da69fff8b75 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:38.143798 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021242c1-a070-4870-a65e-7da69fff8b75 service neutron] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:38.144071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021242c1-a070-4870-a65e-7da69fff8b75 service neutron] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:38.144337 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021242c1-a070-4870-a65e-7da69fff8b75 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:38.153033 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021242c1-a070-4870-a65e-7da69fff8b75 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:38.154474 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021242c1-a070-4870-a65e-7da69fff8b75 service neutron] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:38.163777 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1435/2887] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:16:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:16:38.572303 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5765d6b6-0f97-4694-925b-e93abf444d8c None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:38.573874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-5765d6b6-0f97-4694-925b-e93abf444d8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:38.585840 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5765d6b6-0f97-4694-925b-e93abf444d8c None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:38.590667 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120cacd7-fc6b-4743-9b5a-6c50990f87bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:38.591304 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120cacd7-fc6b-4743-9b5a-6c50990f87bd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:38.591583 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120cacd7-fc6b-4743-9b5a-6c50990f87bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:38.591852 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120cacd7-fc6b-4743-9b5a-6c50990f87bd admin admin] PATH_INFO: `/v3/projects/87b469c9013e44e4af6452e516cf97e2` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:38.592978 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120cacd7-fc6b-4743-9b5a-6c50990f87bd admin admin] RBAC: Authorizing `identity:delete_project(project_id=87b469c9013e44e4af6452e516cf97e2)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:38.594937 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120cacd7-fc6b-4743-9b5a-6c50990f87bd admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:38.705493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-120cacd7-fc6b-4743-9b5a-6c50990f87bd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '87b469c9013e44e4af6452e516cf97e2', 'request_id': 'req-120cacd7-fc6b-4743-9b5a-6c50990f87bd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:16:38.802224 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-120cacd7-fc6b-4743-9b5a-6c50990f87bd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '87b469c9013e44e4af6452e516cf97e2', 'request_id': 'req-120cacd7-fc6b-4743-9b5a-6c50990f87bd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:16:38.808425 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-120cacd7-fc6b-4743-9b5a-6c50990f87bd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '87b469c9013e44e4af6452e516cf97e2', 'request_id': 'req-120cacd7-fc6b-4743-9b5a-6c50990f87bd'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:16:38.819806 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-120cacd7-fc6b-4743-9b5a-6c50990f87bd admin admin] The token cache is being invalidate because project 87b469c9013e44e4af6452e516cf97e2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:16:38.820091 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-120cacd7-fc6b-4743-9b5a-6c50990f87bd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:16:38.820735 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-120cacd7-fc6b-4743-9b5a-6c50990f87bd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:16:38.822199 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1453/2888] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:16:38 2022] DELETE /identity/v3/projects/87b469c9013e44e4af6452e516cf97e2 => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:16:38.827688 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c17382-2f39-432f-badf-8e387928ffdb None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:38.828153 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c17382-2f39-432f-badf-8e387928ffdb None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:38.828264 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c17382-2f39-432f-badf-8e387928ffdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:38.847776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-b4c17382-2f39-432f-badf-8e387928ffdb None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:16:38.848968 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-b4c17382-2f39-432f-badf-8e387928ffdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:38.866299 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:16:38.907497 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1436/2889] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:16:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:16:38.915968 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068b1995-b9d9-4a3e-9415-391bb050fec8 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:38.916247 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068b1995-b9d9-4a3e-9415-391bb050fec8 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:38.916763 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068b1995-b9d9-4a3e-9415-391bb050fec8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:38.936448 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-068b1995-b9d9-4a3e-9415-391bb050fec8 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:16:38.937639 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-068b1995-b9d9-4a3e-9415-391bb050fec8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:38.945666 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1454/2890] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:16:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:16:38.949824 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c17382-2f39-432f-badf-8e387928ffdb None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:38.959462 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c17382-2f39-432f-badf-8e387928ffdb None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:38.964501 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21e4896-1eb7-45a9-b171-7f64820a4340 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:38.965093 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21e4896-1eb7-45a9-b171-7f64820a4340 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:38.965377 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21e4896-1eb7-45a9-b171-7f64820a4340 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:38.965648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21e4896-1eb7-45a9-b171-7f64820a4340 admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:38.966287 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21e4896-1eb7-45a9-b171-7f64820a4340 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:16:38.966628 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21e4896-1eb7-45a9-b171-7f64820a4340 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:38.968645 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21e4896-1eb7-45a9-b171-7f64820a4340 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:38.981197 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1437/2891] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:16:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:16:38.992443 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068b1995-b9d9-4a3e-9415-391bb050fec8 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:39.001617 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068b1995-b9d9-4a3e-9415-391bb050fec8 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:39.006223 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c7cc37-4b9f-44b3-b908-50ee2a46635b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:39.006874 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c7cc37-4b9f-44b3-b908-50ee2a46635b admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:39.007151 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c7cc37-4b9f-44b3-b908-50ee2a46635b admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:39.007423 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c7cc37-4b9f-44b3-b908-50ee2a46635b admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:39.008052 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c7cc37-4b9f-44b3-b908-50ee2a46635b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:39.009944 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c7cc37-4b9f-44b3-b908-50ee2a46635b admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:39.014546 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:16:39.117419 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1455/2892] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:16:38 2022] POST /identity/v3/projects => generated 381 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:16:39.121633 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21e4896-1eb7-45a9-b171-7f64820a4340 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:39.132356 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21e4896-1eb7-45a9-b171-7f64820a4340 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:39.189103 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6148b17d-9231-4977-982d-55d1845361cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:39.189697 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6148b17d-9231-4977-982d-55d1845361cb admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:39.189973 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6148b17d-9231-4977-982d-55d1845361cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:39.190242 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6148b17d-9231-4977-982d-55d1845361cb admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:39.190894 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6148b17d-9231-4977-982d-55d1845361cb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:39.192786 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6148b17d-9231-4977-982d-55d1845361cb admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:39.296844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1438/2893] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:16:39 2022] POST /identity/v3/users => generated 291 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:16:39.301007 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c7cc37-4b9f-44b3-b908-50ee2a46635b admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:39.311951 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c7cc37-4b9f-44b3-b908-50ee2a46635b admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:39.316727 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd41932-7028-4fea-b0c9-f4b6c574a9db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:39.317349 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd41932-7028-4fea-b0c9-f4b6c574a9db admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:39.317627 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd41932-7028-4fea-b0c9-f4b6c574a9db admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:39.317893 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd41932-7028-4fea-b0c9-f4b6c574a9db admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:39.318580 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd41932-7028-4fea-b0c9-f4b6c574a9db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:39.320693 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd41932-7028-4fea-b0c9-f4b6c574a9db admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:39.330524 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1456/2894] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:16:39 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:16:39.334395 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6148b17d-9231-4977-982d-55d1845361cb admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:39.345144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6148b17d-9231-4977-982d-55d1845361cb admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:39.350097 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea03dc8a-81aa-40a8-9620-4f343a0b87d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:39.351099 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea03dc8a-81aa-40a8-9620-4f343a0b87d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:39.351382 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea03dc8a-81aa-40a8-9620-4f343a0b87d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:39.351651 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea03dc8a-81aa-40a8-9620-4f343a0b87d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:39.352365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea03dc8a-81aa-40a8-9620-4f343a0b87d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:39.353969 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea03dc8a-81aa-40a8-9620-4f343a0b87d4 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:39.364860 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1439/2895] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:16:39 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:16:39.369075 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd41932-7028-4fea-b0c9-f4b6c574a9db admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:39.379054 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd41932-7028-4fea-b0c9-f4b6c574a9db admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:39.384111 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa526188-12c3-48e9-8c50-c9ab11180561 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:39.384746 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa526188-12c3-48e9-8c50-c9ab11180561 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:39.385024 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa526188-12c3-48e9-8c50-c9ab11180561 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:39.385317 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa526188-12c3-48e9-8c50-c9ab11180561 admin admin] PATH_INFO: `/v3/projects/01e7888a20c74078b4cb6cb8d6d44899/users/5da7f8c9045041a784d22edfe323716c/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:39.400462 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa526188-12c3-48e9-8c50-c9ab11180561 admin admin] RBAC: Authorizing `identity:create_grant(project_id=01e7888a20c74078b4cb6cb8d6d44899, user_id=5da7f8c9045041a784d22edfe323716c, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:39.403131 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa526188-12c3-48e9-8c50-c9ab11180561 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:39.493537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1457/2896] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:16:39 2022] PUT /identity/v3/projects/01e7888a20c74078b4cb6cb8d6d44899/users/5da7f8c9045041a784d22edfe323716c/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:16:39.498558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea64e52-9309-47b6-82af-15614dfbe253 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:39.498873 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea64e52-9309-47b6-82af-15614dfbe253 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:39.499147 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea64e52-9309-47b6-82af-15614dfbe253 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:39.518182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-aea64e52-9309-47b6-82af-15614dfbe253 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:16:39.519410 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-aea64e52-9309-47b6-82af-15614dfbe253 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:39.577164 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1440/2897] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:16:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:16:39.586652 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa526188-12c3-48e9-8c50-c9ab11180561 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:39.588372 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-fa526188-12c3-48e9-8c50-c9ab11180561 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:39.645382 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa526188-12c3-48e9-8c50-c9ab11180561 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:39.649047 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933dfdfa-e2eb-4837-96f8-8f337235296a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:39.649642 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933dfdfa-e2eb-4837-96f8-8f337235296a service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:39.649921 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933dfdfa-e2eb-4837-96f8-8f337235296a service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:39.650185 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933dfdfa-e2eb-4837-96f8-8f337235296a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:39.658930 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933dfdfa-e2eb-4837-96f8-8f337235296a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:39.660336 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933dfdfa-e2eb-4837-96f8-8f337235296a service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:39.670072 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1458/2898] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:16:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:16:40.982118 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fcb371-0fa3-4e86-9ffc-e42fbeea1239 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:40.982700 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fcb371-0fa3-4e86-9ffc-e42fbeea1239 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:40.982700 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fcb371-0fa3-4e86-9ffc-e42fbeea1239 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:41.012761 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-31fcb371-0fa3-4e86-9ffc-e42fbeea1239 None None] MFA Rules not processed for user `5da7f8c9045041a784d22edfe323716c`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:16:41.014057 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-31fcb371-0fa3-4e86-9ffc-e42fbeea1239 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:41.031796 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:16:41.081386 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1441/2899] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:16:40 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:16:41.093659 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933dfdfa-e2eb-4837-96f8-8f337235296a service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:41.095112 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-933dfdfa-e2eb-4837-96f8-8f337235296a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:41.151679 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933dfdfa-e2eb-4837-96f8-8f337235296a service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:41.155271 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fe22c0-126c-4ec2-85ec-c09fac2c97e3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:41.155866 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fe22c0-126c-4ec2-85ec-c09fac2c97e3 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:41.156148 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fe22c0-126c-4ec2-85ec-c09fac2c97e3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:41.156420 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41fe22c0-126c-4ec2-85ec-c09fac2c97e3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:41.165178 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41fe22c0-126c-4ec2-85ec-c09fac2c97e3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:41.166913 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41fe22c0-126c-4ec2-85ec-c09fac2c97e3 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:41.175508 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1459/2900] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:16:41 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:16:42.830947 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcd5f00-086a-46ee-b166-250fbedff9fe None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:42.831445 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcd5f00-086a-46ee-b166-250fbedff9fe None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:42.831550 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcd5f00-086a-46ee-b166-250fbedff9fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:42.850545 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-cfcd5f00-086a-46ee-b166-250fbedff9fe None None] MFA Rules not processed for user `5da7f8c9045041a784d22edfe323716c`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:16:42.853069 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-cfcd5f00-086a-46ee-b166-250fbedff9fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:42.859929 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1442/2901] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:16:42 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:16:43.102911 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fe22c0-126c-4ec2-85ec-c09fac2c97e3 service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:16:43.104324 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-41fe22c0-126c-4ec2-85ec-c09fac2c97e3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:43.116135 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41fe22c0-126c-4ec2-85ec-c09fac2c97e3 service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:16:43.119752 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47caa73b-7a05-48bd-b1f9-06f4deb67083 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:16:43.120336 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47caa73b-7a05-48bd-b1f9-06f4deb67083 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:16:43.120635 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47caa73b-7a05-48bd-b1f9-06f4deb67083 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:16:43.120905 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47caa73b-7a05-48bd-b1f9-06f4deb67083 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:16:43.122605 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-47caa73b-7a05-48bd-b1f9-06f4deb67083 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:16:43.131948 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47caa73b-7a05-48bd-b1f9-06f4deb67083 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:16:43.134708 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47caa73b-7a05-48bd-b1f9-06f4deb67083 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:16:43.142715 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1460/2902] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:16:43 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:16:52.845139 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=22 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:16:53.139109 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=79,keystone:INSERT=2 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:17:12.999355 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcd5f00-086a-46ee-b166-250fbedff9fe None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:17:13.009111 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcd5f00-086a-46ee-b166-250fbedff9fe None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:17:13.012672 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa20d58d-446c-422a-b860-b62c46173477 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:17:13.013265 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa20d58d-446c-422a-b860-b62c46173477 service cinder] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:17:13.013783 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa20d58d-446c-422a-b860-b62c46173477 service cinder] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:17:13.014056 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa20d58d-446c-422a-b860-b62c46173477 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:17:13.015812 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-aa20d58d-446c-422a-b860-b62c46173477 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:17:13.087446 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa20d58d-446c-422a-b860-b62c46173477 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:17:13.088868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa20d58d-446c-422a-b860-b62c46173477 service cinder] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:17:13.100473 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1443/2903] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:17:12 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:17:23.094714 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=34 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:17:38.702277 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47caa73b-7a05-48bd-b1f9-06f4deb67083 service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:17:38.703713 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-47caa73b-7a05-48bd-b1f9-06f4deb67083 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:17:38.715370 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47caa73b-7a05-48bd-b1f9-06f4deb67083 service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:17:38.719217 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888e3b0d-bbef-43d9-abe9-3e025756b5a4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fae68ca0754a4b0b88a57f39efa2fb45', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:17:38.719800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888e3b0d-bbef-43d9-abe9-3e025756b5a4 service cinder] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:17:38.720074 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888e3b0d-bbef-43d9-abe9-3e025756b5a4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:17:38.720340 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888e3b0d-bbef-43d9-abe9-3e025756b5a4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:17:38.722001 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-888e3b0d-bbef-43d9-abe9-3e025756b5a4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:17:38.735760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888e3b0d-bbef-43d9-abe9-3e025756b5a4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:17:38.737152 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888e3b0d-bbef-43d9-abe9-3e025756b5a4 service cinder] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:17:38.748958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1461/2904] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:17:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:17:48.743518 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=6 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:17:56.431494 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb3cbc3-7c7c-4828-b0e7-baa889bf09ba None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:17:56.431987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb3cbc3-7c7c-4828-b0e7-baa889bf09ba None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:17:56.432098 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb3cbc3-7c7c-4828-b0e7-baa889bf09ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:17:56.451288 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-5fb3cbc3-7c7c-4828-b0e7-baa889bf09ba None None] MFA Rules not processed for user `5da7f8c9045041a784d22edfe323716c`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:17:56.452489 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-5fb3cbc3-7c7c-4828-b0e7-baa889bf09ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:17:56.459277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1444/2905] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:17:56 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:18:00.213061 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888e3b0d-bbef-43d9-abe9-3e025756b5a4 service cinder] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:18:00.222898 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888e3b0d-bbef-43d9-abe9-3e025756b5a4 service cinder] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:18:00.226963 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8da1749-2a21-4a7d-83ea-3bc72a562806 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:18:00.227554 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8da1749-2a21-4a7d-83ea-3bc72a562806 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:18:00.227834 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8da1749-2a21-4a7d-83ea-3bc72a562806 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:18:00.228113 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8da1749-2a21-4a7d-83ea-3bc72a562806 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:18:00.229838 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d8da1749-2a21-4a7d-83ea-3bc72a562806 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:18:00.239975 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8da1749-2a21-4a7d-83ea-3bc72a562806 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:18:00.241393 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8da1749-2a21-4a7d-83ea-3bc72a562806 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:18:00.249996 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1462/2906] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:18:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:18:06.445723 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=3 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:18:10.246293 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=6 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:19:02.654360 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb3cbc3-7c7c-4828-b0e7-baa889bf09ba None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:02.664381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb3cbc3-7c7c-4828-b0e7-baa889bf09ba None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:02.669297 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46062c2a-8772-4241-bcfb-3ea047171417 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:02.670070 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46062c2a-8772-4241-bcfb-3ea047171417 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:02.670378 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46062c2a-8772-4241-bcfb-3ea047171417 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:02.670658 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46062c2a-8772-4241-bcfb-3ea047171417 admin admin] PATH_INFO: `/v3/users/5da7f8c9045041a784d22edfe323716c` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:02.671890 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46062c2a-8772-4241-bcfb-3ea047171417 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5da7f8c9045041a784d22edfe323716c)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:02.673684 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46062c2a-8772-4241-bcfb-3ea047171417 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:02.870159 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-46062c2a-8772-4241-bcfb-3ea047171417 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5da7f8c9045041a784d22edfe323716c'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:19:02.947329 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-46062c2a-8772-4241-bcfb-3ea047171417 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5da7f8c9045041a784d22edfe323716c'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:19:02.947961 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-46062c2a-8772-4241-bcfb-3ea047171417 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5da7f8c9045041a784d22edfe323716c'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:19:02.961265 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-46062c2a-8772-4241-bcfb-3ea047171417 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5da7f8c9045041a784d22edfe323716c'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:19:02.969557 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-46062c2a-8772-4241-bcfb-3ea047171417 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5da7f8c9045041a784d22edfe323716c'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:19:02.976177 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1445/2907] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:19:02 2022] DELETE /identity/v3/users/5da7f8c9045041a784d22edfe323716c => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:19:03.228795 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8da1749-2a21-4a7d-83ea-3bc72a562806 service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:03.243189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8da1749-2a21-4a7d-83ea-3bc72a562806 service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:03.315549 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe8d0d4-fc4b-4279-8922-1b4cac5b145f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:03.316189 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe8d0d4-fc4b-4279-8922-1b4cac5b145f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:03.316471 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe8d0d4-fc4b-4279-8922-1b4cac5b145f admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:03.316788 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe8d0d4-fc4b-4279-8922-1b4cac5b145f admin admin] PATH_INFO: `/v3/projects/01e7888a20c74078b4cb6cb8d6d44899` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:03.317947 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe8d0d4-fc4b-4279-8922-1b4cac5b145f admin admin] RBAC: Authorizing `identity:delete_project(project_id=01e7888a20c74078b4cb6cb8d6d44899)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:03.319941 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe8d0d4-fc4b-4279-8922-1b4cac5b145f admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:03.428868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ffe8d0d4-fc4b-4279-8922-1b4cac5b145f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '01e7888a20c74078b4cb6cb8d6d44899', 'request_id': 'req-ffe8d0d4-fc4b-4279-8922-1b4cac5b145f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:19:03.495886 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ffe8d0d4-fc4b-4279-8922-1b4cac5b145f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '01e7888a20c74078b4cb6cb8d6d44899', 'request_id': 'req-ffe8d0d4-fc4b-4279-8922-1b4cac5b145f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:19:03.503448 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ffe8d0d4-fc4b-4279-8922-1b4cac5b145f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '01e7888a20c74078b4cb6cb8d6d44899', 'request_id': 'req-ffe8d0d4-fc4b-4279-8922-1b4cac5b145f'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:19:03.515013 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ffe8d0d4-fc4b-4279-8922-1b4cac5b145f admin admin] The token cache is being invalidate because project 01e7888a20c74078b4cb6cb8d6d44899 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:19:03.515295 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ffe8d0d4-fc4b-4279-8922-1b4cac5b145f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:19:03.515952 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ffe8d0d4-fc4b-4279-8922-1b4cac5b145f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:19:03.517541 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1463/2908] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:19:03 2022] DELETE /identity/v3/projects/01e7888a20c74078b4cb6cb8d6d44899 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:19:03.522708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4b02a0-510e-4139-a53c-0b5f0c06d0b9 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:03.522993 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4b02a0-510e-4139-a53c-0b5f0c06d0b9 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:03.523273 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4b02a0-510e-4139-a53c-0b5f0c06d0b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:03.542847 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-ff4b02a0-510e-4139-a53c-0b5f0c06d0b9 None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:19:03.544554 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-ff4b02a0-510e-4139-a53c-0b5f0c06d0b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:19:03.605250 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1446/2909] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:19:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:19:03.614643 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8be2290-54fd-4bef-9cc6-f2d76a9c327c None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:03.615033 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8be2290-54fd-4bef-9cc6-f2d76a9c327c None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:03.615380 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8be2290-54fd-4bef-9cc6-f2d76a9c327c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:03.635981 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-a8be2290-54fd-4bef-9cc6-f2d76a9c327c None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:19:03.637210 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a8be2290-54fd-4bef-9cc6-f2d76a9c327c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:19:03.645511 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1464/2910] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:19:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:19:03.649800 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4b02a0-510e-4139-a53c-0b5f0c06d0b9 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:03.660416 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4b02a0-510e-4139-a53c-0b5f0c06d0b9 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:03.665255 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04935693-312a-4144-9d3e-478560eaa0c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:03.665856 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04935693-312a-4144-9d3e-478560eaa0c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:03.666132 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04935693-312a-4144-9d3e-478560eaa0c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:03.666438 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04935693-312a-4144-9d3e-478560eaa0c9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:03.667129 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04935693-312a-4144-9d3e-478560eaa0c9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81569) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 23 14:19:03.667427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04935693-312a-4144-9d3e-478560eaa0c9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:03.669868 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04935693-312a-4144-9d3e-478560eaa0c9 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:03.682444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1447/2911] 10.222.0.62 () {58 vars in 1153 bytes} [Fri Sep 23 14:19:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 23 14:19:03.686842 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8be2290-54fd-4bef-9cc6-f2d76a9c327c None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:03.696600 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8be2290-54fd-4bef-9cc6-f2d76a9c327c None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:03.701424 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3596430-82a2-4aff-b75a-baadfc49672e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:03.702046 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3596430-82a2-4aff-b75a-baadfc49672e admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:03.702324 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3596430-82a2-4aff-b75a-baadfc49672e admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:03.702629 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3596430-82a2-4aff-b75a-baadfc49672e admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:03.703282 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3596430-82a2-4aff-b75a-baadfc49672e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:03.705199 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3596430-82a2-4aff-b75a-baadfc49672e admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:03.812627 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1465/2912] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:19:03 2022] POST /identity/v3/projects => generated 383 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:19:03.816116 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04935693-312a-4144-9d3e-478560eaa0c9 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:03.826543 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04935693-312a-4144-9d3e-478560eaa0c9 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:03.883937 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce5d3ae-4f32-42af-aafe-3cf21192059c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:03.884660 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce5d3ae-4f32-42af-aafe-3cf21192059c admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:03.885031 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce5d3ae-4f32-42af-aafe-3cf21192059c admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:03.885406 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce5d3ae-4f32-42af-aafe-3cf21192059c admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:03.886125 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce5d3ae-4f32-42af-aafe-3cf21192059c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:03.888507 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce5d3ae-4f32-42af-aafe-3cf21192059c admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:03.896285 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:19:04.016831 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1448/2913] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:19:03 2022] POST /identity/v3/users => generated 292 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:19:04.021094 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3596430-82a2-4aff-b75a-baadfc49672e admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:04.032132 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3596430-82a2-4aff-b75a-baadfc49672e admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:04.036915 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deaaa99f-0bee-467d-bba3-d1ffda591b00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:04.037510 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaaa99f-0bee-467d-bba3-d1ffda591b00 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:04.037800 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaaa99f-0bee-467d-bba3-d1ffda591b00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:04.038370 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaaa99f-0bee-467d-bba3-d1ffda591b00 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:04.039012 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deaaa99f-0bee-467d-bba3-d1ffda591b00 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:04.040690 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deaaa99f-0bee-467d-bba3-d1ffda591b00 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:04.050869 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1466/2914] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:19:04 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 23 14:19:04.055556 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce5d3ae-4f32-42af-aafe-3cf21192059c admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:04.065823 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce5d3ae-4f32-42af-aafe-3cf21192059c admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:04.070309 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7152a14a-7b54-443a-96d3-d995c132c2e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:04.070925 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7152a14a-7b54-443a-96d3-d995c132c2e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:04.071205 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7152a14a-7b54-443a-96d3-d995c132c2e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:04.071494 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7152a14a-7b54-443a-96d3-d995c132c2e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:04.072121 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7152a14a-7b54-443a-96d3-d995c132c2e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:04.073719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7152a14a-7b54-443a-96d3-d995c132c2e6 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:04.084932 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1449/2915] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:19:04 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:19:04.093070 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deaaa99f-0bee-467d-bba3-d1ffda591b00 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:04.103096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deaaa99f-0bee-467d-bba3-d1ffda591b00 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:04.107696 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945ef2cd-b49a-4fa4-b1cf-594f6cbec665 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:04.108326 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945ef2cd-b49a-4fa4-b1cf-594f6cbec665 admin admin] REQUEST_METHOD: `PUT` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:04.108603 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945ef2cd-b49a-4fa4-b1cf-594f6cbec665 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:04.108871 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945ef2cd-b49a-4fa4-b1cf-594f6cbec665 admin admin] PATH_INFO: `/v3/projects/6ac60e3f01644a258f0a7ebddf8f4d30/users/6e2be66df7b14b18b55bea07799abab3/roles/e8367770d3124f2db1821b6af66bfeab` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:04.124644 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-945ef2cd-b49a-4fa4-b1cf-594f6cbec665 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ac60e3f01644a258f0a7ebddf8f4d30, user_id=6e2be66df7b14b18b55bea07799abab3, role_id=e8367770d3124f2db1821b6af66bfeab)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:04.127344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-945ef2cd-b49a-4fa4-b1cf-594f6cbec665 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:04.233217 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1467/2916] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:19:04 2022] PUT /identity/v3/projects/6ac60e3f01644a258f0a7ebddf8f4d30/users/6e2be66df7b14b18b55bea07799abab3/roles/e8367770d3124f2db1821b6af66bfeab => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:19:04.238777 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59082e88-9531-48b2-bb4b-52614737450a None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:04.239072 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59082e88-9531-48b2-bb4b-52614737450a None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:04.239349 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59082e88-9531-48b2-bb4b-52614737450a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:04.259176 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-59082e88-9531-48b2-bb4b-52614737450a None None] MFA Rules not processed for user `2ea3af238714467ab48e451a5210c19f`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:19:04.260404 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-59082e88-9531-48b2-bb4b-52614737450a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:19:04.321198 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1450/2917] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:19:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:19:04.331394 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945ef2cd-b49a-4fa4-b1cf-594f6cbec665 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:04.334481 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-945ef2cd-b49a-4fa4-b1cf-594f6cbec665 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:19:04.364781 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:19:04.392684 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945ef2cd-b49a-4fa4-b1cf-594f6cbec665 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:04.396402 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc25bc4-24fb-45e2-b716-ac311f3c30e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:04.397025 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc25bc4-24fb-45e2-b716-ac311f3c30e4 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:04.397518 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc25bc4-24fb-45e2-b716-ac311f3c30e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:04.398023 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc25bc4-24fb-45e2-b716-ac311f3c30e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:04.409097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc25bc4-24fb-45e2-b716-ac311f3c30e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:04.410571 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc25bc4-24fb-45e2-b716-ac311f3c30e4 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:04.420976 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1468/2918] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:19:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:19:09.753672 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc5bbaf-bb0f-4d01-b258-d8669a235922 None None] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:09.754267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc5bbaf-bb0f-4d01-b258-d8669a235922 None None] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:09.754267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc5bbaf-bb0f-4d01-b258-d8669a235922 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:09.786267 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.auth.core [None req-8cc5bbaf-bb0f-4d01-b258-d8669a235922 None None] MFA Rules not processed for user `6e2be66df7b14b18b55bea07799abab3`. Rule list: `[]` (Enabled: `True`). {{(pid=81569) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:19:09.787537 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8cc5bbaf-bb0f-4d01-b258-d8669a235922 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:19:09.863221 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1451/2919] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:19:09 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:19:09.869931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc25bc4-24fb-45e2-b716-ac311f3c30e4 service neutron] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:09.885096 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc25bc4-24fb-45e2-b716-ac311f3c30e4 service neutron] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:09.890593 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83f0a21-a2d2-464c-946c-95759a25a0e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:09.891218 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83f0a21-a2d2-464c-946c-95759a25a0e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:09.891493 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83f0a21-a2d2-464c-946c-95759a25a0e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:09.891760 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83f0a21-a2d2-464c-946c-95759a25a0e0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:09.892387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83f0a21-a2d2-464c-946c-95759a25a0e0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:09.894268 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83f0a21-a2d2-464c-946c-95759a25a0e0 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:09.997969 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1469/2920] 10.222.0.62 () {60 vars in 1153 bytes} [Fri Sep 23 14:19:09 2022] POST /identity/v3/projects => generated 385 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 23 14:19:10.002397 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc5bbaf-bb0f-4d01-b258-d8669a235922 None None] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:10.013291 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc5bbaf-bb0f-4d01-b258-d8669a235922 None None] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:10.081052 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c2ffb7-5c20-4b91-b658-6ffb45bf84c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:10.081921 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c2ffb7-5c20-4b91-b658-6ffb45bf84c2 admin admin] REQUEST_METHOD: `POST` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:10.082245 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c2ffb7-5c20-4b91-b658-6ffb45bf84c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:10.082639 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c2ffb7-5c20-4b91-b658-6ffb45bf84c2 admin admin] PATH_INFO: `/v3/users` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:10.083329 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c2ffb7-5c20-4b91-b658-6ffb45bf84c2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:10.085469 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c2ffb7-5c20-4b91-b658-6ffb45bf84c2 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:10.175573 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:19:10.181807 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1452/2921] 10.222.0.62 () {60 vars in 1144 bytes} [Fri Sep 23 14:19:10 2022] POST /identity/v3/users => generated 299 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 23 14:19:10.185908 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83f0a21-a2d2-464c-946c-95759a25a0e0 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:10.197696 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83f0a21-a2d2-464c-946c-95759a25a0e0 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:10.205827 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a124d789-c068-43ae-93e7-d6aef32860d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:10.206522 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a124d789-c068-43ae-93e7-d6aef32860d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:10.206807 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a124d789-c068-43ae-93e7-d6aef32860d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:10.207092 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a124d789-c068-43ae-93e7-d6aef32860d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:10.207745 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a124d789-c068-43ae-93e7-d6aef32860d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:10.209390 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a124d789-c068-43ae-93e7-d6aef32860d1 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:10.223805 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1470/2922] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:19:10 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:19:10.228301 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c2ffb7-5c20-4b91-b658-6ffb45bf84c2 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:10.241348 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c2ffb7-5c20-4b91-b658-6ffb45bf84c2 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:10.246023 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b275aa4e-4863-43f3-8d8a-51852ea781aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:10.246722 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b275aa4e-4863-43f3-8d8a-51852ea781aa admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:10.247017 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b275aa4e-4863-43f3-8d8a-51852ea781aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:10.247285 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b275aa4e-4863-43f3-8d8a-51852ea781aa admin admin] PATH_INFO: `/v3/projects/2042b7f74f6940f6a3d990d4ed5f7a18/users/8a330383361f43dfae5ce499d18b1155/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:10.265464 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b275aa4e-4863-43f3-8d8a-51852ea781aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=2042b7f74f6940f6a3d990d4ed5f7a18, user_id=8a330383361f43dfae5ce499d18b1155, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:10.268844 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b275aa4e-4863-43f3-8d8a-51852ea781aa admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:10.369731 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1453/2923] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:19:10 2022] PUT /identity/v3/projects/2042b7f74f6940f6a3d990d4ed5f7a18/users/8a330383361f43dfae5ce499d18b1155/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:19:10.379957 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a124d789-c068-43ae-93e7-d6aef32860d1 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:10.391397 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a124d789-c068-43ae-93e7-d6aef32860d1 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:10.457626 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb081ccf-7b65-49f7-b7d7-7c9d976ab250 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:10.458492 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb081ccf-7b65-49f7-b7d7-7c9d976ab250 admin admin] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:10.458775 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb081ccf-7b65-49f7-b7d7-7c9d976ab250 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:10.459041 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb081ccf-7b65-49f7-b7d7-7c9d976ab250 admin admin] PATH_INFO: `/v3/roles` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:10.459691 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb081ccf-7b65-49f7-b7d7-7c9d976ab250 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:10.461302 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb081ccf-7b65-49f7-b7d7-7c9d976ab250 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:10.476861 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1471/2924] 10.222.0.62 () {58 vars in 1122 bytes} [Fri Sep 23 14:19:10 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 23 14:19:10.478002 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b275aa4e-4863-43f3-8d8a-51852ea781aa admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:10.489332 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b275aa4e-4863-43f3-8d8a-51852ea781aa admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:10.494073 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f86bb5-b494-46fe-9746-4023b7052548 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:10.494769 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f86bb5-b494-46fe-9746-4023b7052548 admin admin] REQUEST_METHOD: `PUT` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:10.495044 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f86bb5-b494-46fe-9746-4023b7052548 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:10.495316 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f86bb5-b494-46fe-9746-4023b7052548 admin admin] PATH_INFO: `/v3/projects/2042b7f74f6940f6a3d990d4ed5f7a18/users/8a330383361f43dfae5ce499d18b1155/roles/727f0fa598e54b9a929d2cf6277c875e` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:10.497726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f86bb5-b494-46fe-9746-4023b7052548 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2042b7f74f6940f6a3d990d4ed5f7a18, user_id=8a330383361f43dfae5ce499d18b1155, role_id=727f0fa598e54b9a929d2cf6277c875e)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:10.500639 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f86bb5-b494-46fe-9746-4023b7052548 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:10.511987 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1454/2925] 10.222.0.62 () {60 vars in 1483 bytes} [Fri Sep 23 14:19:10 2022] PUT /identity/v3/projects/2042b7f74f6940f6a3d990d4ed5f7a18/users/8a330383361f43dfae5ce499d18b1155/roles/727f0fa598e54b9a929d2cf6277c875e => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:19:16.337431 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b877b0b-45fc-4072-835a-18df4a2d5777 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:16.338272 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b877b0b-45fc-4072-835a-18df4a2d5777 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:16.338802 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b877b0b-45fc-4072-835a-18df4a2d5777 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:16.373831 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-1b877b0b-45fc-4072-835a-18df4a2d5777 None None] MFA Rules not processed for user `8a330383361f43dfae5ce499d18b1155`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:19:16.375476 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-1b877b0b-45fc-4072-835a-18df4a2d5777 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:19:16.464484 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1472/2926] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:19:16 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:19:16.478144 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f86bb5-b494-46fe-9746-4023b7052548 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:16.479941 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-38f86bb5-b494-46fe-9746-4023b7052548 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:19:16.545510 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f86bb5-b494-46fe-9746-4023b7052548 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:16.549475 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dde5530-41f4-4c75-966d-ff053d80f5fb service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'acdf4a7bd608492390176dcc42ca1267', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:16.550119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dde5530-41f4-4c75-966d-ff053d80f5fb service glance] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:16.550433 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dde5530-41f4-4c75-966d-ff053d80f5fb service glance] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:16.550708 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dde5530-41f4-4c75-966d-ff053d80f5fb service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:16.611411 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dde5530-41f4-4c75-966d-ff053d80f5fb service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:16.612985 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dde5530-41f4-4c75-966d-ff053d80f5fb service glance] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:16.622941 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1455/2927] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:19:16 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:19:16.907948 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be508d7b-560b-4781-89be-b19555919d9a None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:16.908238 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be508d7b-560b-4781-89be-b19555919d9a None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:16.908509 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be508d7b-560b-4781-89be-b19555919d9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:16.937628 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:19:16.962007 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-be508d7b-560b-4781-89be-b19555919d9a None None] MFA Rules not processed for user `b471d768cd814ff8b4f6bd35d21e24a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:19:16.963263 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-be508d7b-560b-4781-89be-b19555919d9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:19:17.033435 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1473/2928] 10.222.0.62 () {58 vars in 1009 bytes} [Fri Sep 23 14:19:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:19:17.202787 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dde5530-41f4-4c75-966d-ff053d80f5fb service glance] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:17.204313 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3dde5530-41f4-4c75-966d-ff053d80f5fb service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:19:17.291801 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:19:17.301959 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dde5530-41f4-4c75-966d-ff053d80f5fb service glance] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:17.307596 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e4d665-a8a6-4c32-a2d5-506319b8e94a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b9269509f1d454eab7a3008c2ba0039', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:17.308213 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e4d665-a8a6-4c32-a2d5-506319b8e94a service swift] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:17.308490 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e4d665-a8a6-4c32-a2d5-506319b8e94a service swift] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:17.309369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e4d665-a8a6-4c32-a2d5-506319b8e94a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:17.318776 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e4d665-a8a6-4c32-a2d5-506319b8e94a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:17.320365 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e4d665-a8a6-4c32-a2d5-506319b8e94a service swift] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:17.332137 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1456/2929] 10.222.0.62 () {60 vars in 1384 bytes} [Fri Sep 23 14:19:17 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 23 14:19:27.029551 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=32 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:19:27.466434 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=7 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:19:27.860770 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6833197-4f0d-42a0-a6c2-851a7f2748b3 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:27.861056 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6833197-4f0d-42a0-a6c2-851a7f2748b3 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:27.861331 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6833197-4f0d-42a0-a6c2-851a7f2748b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:27.883667 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-a6833197-4f0d-42a0-a6c2-851a7f2748b3 None None] MFA Rules not processed for user `6e2be66df7b14b18b55bea07799abab3`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:19:27.884870 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a6833197-4f0d-42a0-a6c2-851a7f2748b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:19:27.892245 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1474/2930] 10.222.0.62 () {58 vars in 958 bytes} [Fri Sep 23 14:19:27 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:19:28.578095 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e4d665-a8a6-4c32-a2d5-506319b8e94a service swift] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:28.579656 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-47e4d665-a8a6-4c32-a2d5-506319b8e94a service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:19:28.655064 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e4d665-a8a6-4c32-a2d5-506319b8e94a service swift] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:28.660249 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a21ea64-5aa4-4757-a20f-f6a9871772f1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce249d4a99c34891b6ba941f57a14dd4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:28.660850 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a21ea64-5aa4-4757-a20f-f6a9871772f1 service placement] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:28.661123 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a21ea64-5aa4-4757-a20f-f6a9871772f1 service placement] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:28.661388 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a21ea64-5aa4-4757-a20f-f6a9871772f1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:28.663157 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-4a21ea64-5aa4-4757-a20f-f6a9871772f1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:19:28.675009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a21ea64-5aa4-4757-a20f-f6a9871772f1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:28.676582 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a21ea64-5aa4-4757-a20f-f6a9871772f1 service placement] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:28.687470 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1457/2931] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:19:28 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:19:31.627570 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbb2632-c68a-4ab1-8725-9397228cbac4 None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:31.628101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbb2632-c68a-4ab1-8725-9397228cbac4 None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:31.628101 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbb2632-c68a-4ab1-8725-9397228cbac4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:31.650120 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-5bbb2632-c68a-4ab1-8725-9397228cbac4 None None] MFA Rules not processed for user `b471d768cd814ff8b4f6bd35d21e24a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:19:31.651506 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-5bbb2632-c68a-4ab1-8725-9397228cbac4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:19:31.659261 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1475/2932] 10.222.0.62 () {58 vars in 1009 bytes} [Fri Sep 23 14:19:31 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 23 14:19:31.672741 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a21ea64-5aa4-4757-a20f-f6a9871772f1 service placement] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:31.685368 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a21ea64-5aa4-4757-a20f-f6a9871772f1 service placement] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:31.690478 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dba0921-7c62-40ab-9e32-1bd23cb9ea75 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b9269509f1d454eab7a3008c2ba0039', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:31.691071 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dba0921-7c62-40ab-9e32-1bd23cb9ea75 service swift] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:31.691350 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dba0921-7c62-40ab-9e32-1bd23cb9ea75 service swift] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:31.691615 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dba0921-7c62-40ab-9e32-1bd23cb9ea75 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:31.700185 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dba0921-7c62-40ab-9e32-1bd23cb9ea75 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:31.701753 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dba0921-7c62-40ab-9e32-1bd23cb9ea75 service swift] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:31.710119 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1458/2933] 10.222.0.62 () {60 vars in 1384 bytes} [Fri Sep 23 14:19:31 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 23 14:19:40.456841 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bbb2632-c68a-4ab1-8725-9397228cbac4 None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:40.458439 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-5bbb2632-c68a-4ab1-8725-9397228cbac4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:19:40.534186 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bbb2632-c68a-4ab1-8725-9397228cbac4 None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:40.539886 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a8a379-5e96-4737-9b3d-989ec781d0ba service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:40.540495 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a8a379-5e96-4737-9b3d-989ec781d0ba service nova] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:40.540778 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a8a379-5e96-4737-9b3d-989ec781d0ba service nova] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:40.541046 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a8a379-5e96-4737-9b3d-989ec781d0ba service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:40.542868 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-d5a8a379-5e96-4737-9b3d-989ec781d0ba service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:19:40.560330 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a8a379-5e96-4737-9b3d-989ec781d0ba service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:40.560330 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a8a379-5e96-4737-9b3d-989ec781d0ba service nova] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:40.570011 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1476/2934] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:19:40 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:19:41.706738 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=38 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:19:46.172752 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dba0921-7c62-40ab-9e32-1bd23cb9ea75 service swift] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:19:46.180387 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dba0921-7c62-40ab-9e32-1bd23cb9ea75 service swift] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:19:46.186224 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb2f1da-f254-44dd-a9f2-a24b8e83d695 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:19:46.187280 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb2f1da-f254-44dd-a9f2-a24b8e83d695 service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:19:46.187745 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb2f1da-f254-44dd-a9f2-a24b8e83d695 service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:19:46.188097 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb2f1da-f254-44dd-a9f2-a24b8e83d695 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:19:46.189967 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-4bb2f1da-f254-44dd-a9f2-a24b8e83d695 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:19:46.202321 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb2f1da-f254-44dd-a9f2-a24b8e83d695 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:19:46.204551 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bb2f1da-f254-44dd-a9f2-a24b8e83d695 service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:19:46.220718 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1459/2935] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:19:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:19:50.565387 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=40 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:19:56.215608 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=6 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:20:02.006066 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a8a379-5e96-4737-9b3d-989ec781d0ba service nova] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:20:02.059645 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a8a379-5e96-4737-9b3d-989ec781d0ba service nova] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:20:02.063299 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0bdd11-3b7b-4d91-bdea-7bb2a71a81d8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '580a35627c234a25aa04f5d376177ca5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:20:02.064683 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0bdd11-3b7b-4d91-bdea-7bb2a71a81d8 service neutron] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:20:02.064958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0bdd11-3b7b-4d91-bdea-7bb2a71a81d8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:20:02.065221 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0bdd11-3b7b-4d91-bdea-7bb2a71a81d8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:20:02.074303 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0bdd11-3b7b-4d91-bdea-7bb2a71a81d8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:20:02.075730 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0bdd11-3b7b-4d91-bdea-7bb2a71a81d8 service neutron] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:20:02.086044 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1477/2936] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:20:02 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:20:03.980726 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb2f1da-f254-44dd-a9f2-a24b8e83d695 service nova] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:20:03.993355 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb2f1da-f254-44dd-a9f2-a24b8e83d695 service nova] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:20:04.000444 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342c4e9b-d736-4401-951a-9c9ddacd6dcf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f01ed94dbb384a178ad0a45a6f3d8c17', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'reader', 'ResellerAdmin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:20:04.001676 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342c4e9b-d736-4401-951a-9c9ddacd6dcf service nova] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:20:04.001954 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342c4e9b-d736-4401-951a-9c9ddacd6dcf service nova] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:20:04.002221 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-342c4e9b-d736-4401-951a-9c9ddacd6dcf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:20:04.004091 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-342c4e9b-d736-4401-951a-9c9ddacd6dcf service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:20:04.016686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342c4e9b-d736-4401-951a-9c9ddacd6dcf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:20:04.018253 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-342c4e9b-d736-4401-951a-9c9ddacd6dcf service nova] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:20:04.029758 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1460/2937] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:20:03 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 23 14:20:12.082400 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=26 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:20:14.024427 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=6 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:21:32.143161 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041c4849-4e25-46ce-9d8a-04a1531c435d None None] REQUEST_METHOD: `POST` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:21:32.143700 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041c4849-4e25-46ce-9d8a-04a1531c435d None None] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:21:32.143700 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041c4849-4e25-46ce-9d8a-04a1531c435d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:21:32.184371 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.auth.core [None req-041c4849-4e25-46ce-9d8a-04a1531c435d None None] MFA Rules not processed for user `b471d768cd814ff8b4f6bd35d21e24a8`. Rule list: `[]` (Enabled: `True`). {{(pid=81568) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 23 14:21:32.185573 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-041c4849-4e25-46ce-9d8a-04a1531c435d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:21:32.220690 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1478/2938] 10.222.0.62 () {58 vars in 967 bytes} [Fri Sep 23 14:21:32 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 23 14:21:32.262299 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342c4e9b-d736-4401-951a-9c9ddacd6dcf service nova] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:21:32.304686 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-342c4e9b-d736-4401-951a-9c9ddacd6dcf service nova] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:21:32.309553 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7793fbb2-d36d-445b-bf62-56f7a0a10b09 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4b9269509f1d454eab7a3008c2ba0039', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:21:32.310136 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7793fbb2-d36d-445b-bf62-56f7a0a10b09 service swift] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:21:32.310442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7793fbb2-d36d-445b-bf62-56f7a0a10b09 service swift] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:21:32.310713 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7793fbb2-d36d-445b-bf62-56f7a0a10b09 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:21:32.318810 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7793fbb2-d36d-445b-bf62-56f7a0a10b09 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:21:32.320442 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7793fbb2-d36d-445b-bf62-56f7a0a10b09 service swift] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:21:32.328700 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1461/2939] 10.222.0.62 () {60 vars in 1384 bytes} [Fri Sep 23 14:21:32 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 23 14:21:35.187398 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041c4849-4e25-46ce-9d8a-04a1531c435d None None] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:21:35.189062 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-041c4849-4e25-46ce-9d8a-04a1531c435d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:21:35.233498 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041c4849-4e25-46ce-9d8a-04a1531c435d None None] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:21:35.240597 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22f1c31-ec54-465f-8762-fc02d39319d9 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce249d4a99c34891b6ba941f57a14dd4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:21:35.241344 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22f1c31-ec54-465f-8762-fc02d39319d9 service placement] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:21:35.241629 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22f1c31-ec54-465f-8762-fc02d39319d9 service placement] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:21:35.241920 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22f1c31-ec54-465f-8762-fc02d39319d9 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:21:35.243741 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-a22f1c31-ec54-465f-8762-fc02d39319d9 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:21:35.258097 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22f1c31-ec54-465f-8762-fc02d39319d9 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:21:35.259755 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22f1c31-ec54-465f-8762-fc02d39319d9 service placement] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:21:35.274229 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1479/2940] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:21:35 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:21:37.115029 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7793fbb2-d36d-445b-bf62-56f7a0a10b09 service swift] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:21:37.126646 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7793fbb2-d36d-445b-bf62-56f7a0a10b09 service swift] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:21:37.131756 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee56c3a-1a25-4bbe-a7b5-34a16b913e5f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce249d4a99c34891b6ba941f57a14dd4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:21:37.132369 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee56c3a-1a25-4bbe-a7b5-34a16b913e5f service placement] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:21:37.132648 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee56c3a-1a25-4bbe-a7b5-34a16b913e5f service placement] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:21:37.132922 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee56c3a-1a25-4bbe-a7b5-34a16b913e5f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:21:37.134719 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-bee56c3a-1a25-4bbe-a7b5-34a16b913e5f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:21:37.147364 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee56c3a-1a25-4bbe-a7b5-34a16b913e5f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:21:37.150155 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee56c3a-1a25-4bbe-a7b5-34a16b913e5f service placement] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:21:37.160778 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1462/2941] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:21:37 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:21:41.122603 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22f1c31-ec54-465f-8762-fc02d39319d9 service placement] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:21:41.133407 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22f1c31-ec54-465f-8762-fc02d39319d9 service placement] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:21:41.139575 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5f827b-4555-45d9-a379-ae8f8b52f572 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce249d4a99c34891b6ba941f57a14dd4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:21:41.140474 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5f827b-4555-45d9-a379-ae8f8b52f572 service placement] REQUEST_METHOD: `GET` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:21:41.140912 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5f827b-4555-45d9-a379-ae8f8b52f572 service placement] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:21:41.141192 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5f827b-4555-45d9-a379-ae8f8b52f572 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:21:41.144218 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-4c5f827b-4555-45d9-a379-ae8f8b52f572 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:21:41.159377 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5f827b-4555-45d9-a379-ae8f8b52f572 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:21:41.161997 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5f827b-4555-45d9-a379-ae8f8b52f572 service placement] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:21:41.173786 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1480/2942] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:21:41 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:21:41.276887 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee56c3a-1a25-4bbe-a7b5-34a16b913e5f service placement] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:21:41.366107 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee56c3a-1a25-4bbe-a7b5-34a16b913e5f service placement] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:21:41.370799 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa220ee-b834-4d24-ae0d-f9fae312f434 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:21:41.373042 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa220ee-b834-4d24-ae0d-f9fae312f434 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:21:41.373042 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa220ee-b834-4d24-ae0d-f9fae312f434 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:21:41.373042 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa220ee-b834-4d24-ae0d-f9fae312f434 admin admin] PATH_INFO: `/v3/users/6e2be66df7b14b18b55bea07799abab3` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:21:41.373815 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa220ee-b834-4d24-ae0d-f9fae312f434 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6e2be66df7b14b18b55bea07799abab3)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:21:41.375768 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa220ee-b834-4d24-ae0d-f9fae312f434 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:21:41.562935 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3fa220ee-b834-4d24-ae0d-f9fae312f434 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6e2be66df7b14b18b55bea07799abab3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:21:41.616329 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3fa220ee-b834-4d24-ae0d-f9fae312f434 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6e2be66df7b14b18b55bea07799abab3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:21:41.616951 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3fa220ee-b834-4d24-ae0d-f9fae312f434 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6e2be66df7b14b18b55bea07799abab3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:21:41.630393 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3fa220ee-b834-4d24-ae0d-f9fae312f434 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6e2be66df7b14b18b55bea07799abab3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:21:41.637556 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3fa220ee-b834-4d24-ae0d-f9fae312f434 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6e2be66df7b14b18b55bea07799abab3'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:21:41.644271 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1463/2943] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:21:41 2022] DELETE /identity/v3/users/6e2be66df7b14b18b55bea07799abab3 => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:21:41.909167 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5f827b-4555-45d9-a379-ae8f8b52f572 service placement] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:21:41.922680 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5f827b-4555-45d9-a379-ae8f8b52f572 service placement] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:21:41.990872 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761ecfc3-7cb5-425f-ac2f-91b55c8f7182 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:21:41.991537 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761ecfc3-7cb5-425f-ac2f-91b55c8f7182 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:21:41.991811 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761ecfc3-7cb5-425f-ac2f-91b55c8f7182 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:21:41.992082 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-761ecfc3-7cb5-425f-ac2f-91b55c8f7182 admin admin] PATH_INFO: `/v3/projects/6ac60e3f01644a258f0a7ebddf8f4d30` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:21:41.993255 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761ecfc3-7cb5-425f-ac2f-91b55c8f7182 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6ac60e3f01644a258f0a7ebddf8f4d30)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:21:41.995810 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-761ecfc3-7cb5-425f-ac2f-91b55c8f7182 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:21:42.101783 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-761ecfc3-7cb5-425f-ac2f-91b55c8f7182 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6ac60e3f01644a258f0a7ebddf8f4d30', 'request_id': 'req-761ecfc3-7cb5-425f-ac2f-91b55c8f7182'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:21:42.166876 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-761ecfc3-7cb5-425f-ac2f-91b55c8f7182 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6ac60e3f01644a258f0a7ebddf8f4d30', 'request_id': 'req-761ecfc3-7cb5-425f-ac2f-91b55c8f7182'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:21:42.173743 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-761ecfc3-7cb5-425f-ac2f-91b55c8f7182 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6ac60e3f01644a258f0a7ebddf8f4d30', 'request_id': 'req-761ecfc3-7cb5-425f-ac2f-91b55c8f7182'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:21:42.174958 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:21:42.199622 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-761ecfc3-7cb5-425f-ac2f-91b55c8f7182 admin admin] The token cache is being invalidate because project 6ac60e3f01644a258f0a7ebddf8f4d30 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:21:42.199931 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-761ecfc3-7cb5-425f-ac2f-91b55c8f7182 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:21:42.200552 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-761ecfc3-7cb5-425f-ac2f-91b55c8f7182 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:21:42.202106 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1481/2944] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:21:41 2022] DELETE /identity/v3/projects/6ac60e3f01644a258f0a7ebddf8f4d30 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 23 14:21:42.206566 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa220ee-b834-4d24-ae0d-f9fae312f434 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:21:42.208015 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3fa220ee-b834-4d24-ae0d-f9fae312f434 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:21:42.217447 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:21:42.221565 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa220ee-b834-4d24-ae0d-f9fae312f434 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:21:42.285777 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0f483a-9354-4571-b434-f6fc79932e36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:21:42.286558 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0f483a-9354-4571-b434-f6fc79932e36 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:21:42.286736 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0f483a-9354-4571-b434-f6fc79932e36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:21:42.287009 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0f483a-9354-4571-b434-f6fc79932e36 admin admin] PATH_INFO: `/v3/users/8a330383361f43dfae5ce499d18b1155` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:21:42.288184 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a0f483a-9354-4571-b434-f6fc79932e36 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8a330383361f43dfae5ce499d18b1155)` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:21:42.290235 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a0f483a-9354-4571-b434-f6fc79932e36 admin admin] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:21:42.518025 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3a0f483a-9354-4571-b434-f6fc79932e36 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8a330383361f43dfae5ce499d18b1155'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:21:42.598901 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3a0f483a-9354-4571-b434-f6fc79932e36 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8a330383361f43dfae5ce499d18b1155'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:21:42.599532 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3a0f483a-9354-4571-b434-f6fc79932e36 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8a330383361f43dfae5ce499d18b1155'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:21:42.614182 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3a0f483a-9354-4571-b434-f6fc79932e36 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8a330383361f43dfae5ce499d18b1155'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:21:42.621381 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.notifications [None req-3a0f483a-9354-4571-b434-f6fc79932e36 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8a330383361f43dfae5ce499d18b1155'} {{(pid=81569) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:21:42.627463 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1464/2945] 10.222.0.62 () {58 vars in 1224 bytes} [Fri Sep 23 14:21:42 2022] DELETE /identity/v3/users/8a330383361f43dfae5ce499d18b1155 => generated 0 bytes in 421 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:21:42.911662 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761ecfc3-7cb5-425f-ac2f-91b55c8f7182 admin admin] Authenticating user token {{(pid=81568) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:21:42.913104 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.fernet_utils [None req-761ecfc3-7cb5-425f-ac2f-91b55c8f7182 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81568) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:21:42.927618 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-761ecfc3-7cb5-425f-ac2f-91b55c8f7182 admin admin] Validating token access rules against request {{(pid=81568) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:21:42.989515 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0fa9a4-fafa-47ea-9fb7-8becc10f65a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2ea3af238714467ab48e451a5210c19f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e15d4d7f53c543e88cdf9075fbadb3c8', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81568) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:21:42.990058 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0fa9a4-fafa-47ea-9fb7-8becc10f65a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:21:42.990350 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0fa9a4-fafa-47ea-9fb7-8becc10f65a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:21:42.990638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0fa9a4-fafa-47ea-9fb7-8becc10f65a9 admin admin] PATH_INFO: `/v3/projects/2042b7f74f6940f6a3d990d4ed5f7a18` {{(pid=81568) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:21:42.991749 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0fa9a4-fafa-47ea-9fb7-8becc10f65a9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2042b7f74f6940f6a3d990d4ed5f7a18)` {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:21:42.993638 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0fa9a4-fafa-47ea-9fb7-8becc10f65a9 admin admin] RBAC: Authorization granted {{(pid=81568) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:21:43.128811 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ec0fa9a4-fafa-47ea-9fb7-8becc10f65a9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2042b7f74f6940f6a3d990d4ed5f7a18', 'request_id': 'req-ec0fa9a4-fafa-47ea-9fb7-8becc10f65a9'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:21:43.195310 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ec0fa9a4-fafa-47ea-9fb7-8becc10f65a9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2042b7f74f6940f6a3d990d4ed5f7a18', 'request_id': 'req-ec0fa9a4-fafa-47ea-9fb7-8becc10f65a9'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:21:43.201765 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ec0fa9a4-fafa-47ea-9fb7-8becc10f65a9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2042b7f74f6940f6a3d990d4ed5f7a18', 'request_id': 'req-ec0fa9a4-fafa-47ea-9fb7-8becc10f65a9'} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:21:43.216409 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ec0fa9a4-fafa-47ea-9fb7-8becc10f65a9 admin admin] The token cache is being invalidate because project 2042b7f74f6940f6a3d990d4ed5f7a18 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81568) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 23 14:21:43.222664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ec0fa9a4-fafa-47ea-9fb7-8becc10f65a9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:21:43.222664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG keystone.notifications [None req-ec0fa9a4-fafa-47ea-9fb7-8becc10f65a9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81568) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 23 14:21:43.222664 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: [pid: 81568|app: 0|req: 1482/2946] 10.222.0.62 () {58 vars in 1233 bytes} [Fri Sep 23 14:21:42 2022] DELETE /identity/v3/projects/2042b7f74f6940f6a3d990d4ed5f7a18 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 23 14:21:52.623973 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=51,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:21:53.211936 dv-cinder-857434-4-iscsi devstack@keystone.service[81568]: DEBUG dbcounter [-] [81568] Writing DB stats keystone:SELECT=59,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81568) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 23 14:26:40.263898 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0f483a-9354-4571-b434-f6fc79932e36 admin admin] Authenticating user token {{(pid=81569) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 23 14:26:40.265318 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-3a0f483a-9354-4571-b434-f6fc79932e36 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:26:40.335766 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0f483a-9354-4571-b434-f6fc79932e36 admin admin] Validating token access rules against request {{(pid=81569) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 23 14:26:40.398277 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efd2cd1-178d-40d2-b820-31fc8ea0d6fe service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ce249d4a99c34891b6ba941f57a14dd4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '2a19fd4315ad4a6eb349dfe8cfa66c23', 'project_domain_id': 'default', 'roles': ['service', 'reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81569) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 23 14:26:40.398930 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efd2cd1-178d-40d2-b820-31fc8ea0d6fe service placement] REQUEST_METHOD: `GET` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 23 14:26:40.399209 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efd2cd1-178d-40d2-b820-31fc8ea0d6fe service placement] SCRIPT_NAME: `/identity` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 23 14:26:40.399480 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efd2cd1-178d-40d2-b820-31fc8ea0d6fe service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81569) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 23 14:26:40.401484 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.fernet_utils [None req-8efd2cd1-178d-40d2-b820-31fc8ea0d6fe service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81569) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 23 14:26:40.413619 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efd2cd1-178d-40d2-b820-31fc8ea0d6fe service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 23 14:26:40.415214 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efd2cd1-178d-40d2-b820-31fc8ea0d6fe service placement] RBAC: Authorization granted {{(pid=81569) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 23 14:26:40.425793 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: [pid: 81569|app: 0|req: 1465/2947] 10.222.0.62 () {60 vars in 1365 bytes} [Fri Sep 23 14:26:40 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 23 14:26:50.421210 dv-cinder-857434-4-iscsi devstack@keystone.service[81569]: DEBUG dbcounter [-] [81569] Writing DB stats keystone:SELECT=58 {{(pid=81569) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}